Source of Acquisition 
NASA Contractor/Grantee 


Development and Assessment of a Novel Training Package 
For Basic Maneuvering Tasks on a Flight Simulator 
Using Self Instruction Methods and Above Real Time Training (ARTT) 


FINAL TECHNICAL REPORT 


By 


Syed Firasat Ali (Principal Investigator), M. Javed Khan, Marcia J. Rossi, 
Bruce E. Heath, Peter Crane (Consultant from USAF), Marcus Ward, 
Tomyka Crier, Tremaine Knighten and Christi Culpepper 


Aerospace Science Engineering Department and 
The Department of Psychology and Sociology 
Tuskegee University, Tuskegee, AL 36088 


Sponsored by 

NASA Dryden Flight Research Center 
Grant NAG4 - 226 
(DUNNS# 128214178) 


March 22, 2002 to March 21, 2007 



2. DISTRIBUTION LIST AND DISCLAIMER 


1 NASA Center for Aerospace Information (CAS1) 
Attention: Document Processing Center 

7121 Standard Drive 
Hanover, MD 21076 

2 Administrative Grant Officer (AGO) 

Office of Naval Research (ONR) 

Atlanta Regional Office 

Attn: Regional Director 
100 Alabama Street, Suite 4R15 
Atlanta, GA 30303 

3 Technical Officer (TO) 

Dr. Kajal K. Gupta 

NASA Dryden Flight Research Center 
P. O. Box 273, Mail Stop 2701 
Edwards, CA 93523 

4 Grant Officer (GO) 

Syreeta J. Stewart 

NASA Glenn Research Center 

2100 Brookpark Road, Mail Stop 500-319 

Cleveland, OH 44135-3191 

5 Dr. Miriam Rodon-Naveira 

NASA Dryden Flight Research Center 
P. O. Box 273, Mail Stop 1023 
Edwards, CA 93523 

6 Dr. Lance Richards 

NASA Dryden Flight Research Center 
P. O. Box 273, Mail Stop 4820-1 C 
Edwards, CA 93523 


Any opinions, findings, and conclusions or recommendations 
in this report are those of the authors and 
they do not necessarily reflect the views of the 
National Aeronautics and Space Administration 


The reported work has not resulted in any invention 


3. TABLE OF CONTENTS 


1. COVER PAGE 1 

2. DISTRIBUTION LIST AND DISCLAIMER 2 

3. TABLE OF CONTENTS 3 

4. SUMMARY OF COMPLETE REPORT 5 to 1 0 

5 . ACKNOWLEDGEMENTS FOR THE COMPLETER REPORT 1 0 


6. SIX DIFFERENT PUBLICATIONS IN THE FOLLOWING SEQUENCE 
( They have their own page numbers) 

I. “An Evaluation of Training Interventions and Computed Scoring Techniques 
for Grading a Level Turn Task and a Straight in Landing Approach on a PC- 
Based Flight Simulator”, MS thesis of Bruce E. Heath, Major Professor M. 
Javed Khan, at Tuskegee University, May 2007, 76 pages. 

II. “An Experimental Study of the Effect of Out-of-the-Window Cues on 
Training of Novice Pilots on a Flight Simulator” by M. Javed Khan, Marcia 
Rossi, Bruce Heath, S. Firasat Ali, Marcus Ward, Proceedings, Human 
Factors and Ergonomics Society Conference, San Francisco, California, 
October 2006, 10 pages. 

III. “Correlating Automated and Flight Instructor Assessments of Straight-In 
Landing Approaches by Novice Pilots on a Flight Simulator” by Bruce E. 
Heath, M. Javed Khan, Marcia Rossi, Syed Firasat Ali, Proceedings, 
Huntsville Simulation Conference, Huntsville, Alabama, October 2005, 10 
pages. 

IV. Using a Low Cost Flight Simulation Environment for Interdisciplinary 
Education by M. Javed Khan, Marcia Rossi, Syed F. Ali, Proceedings, ASEE 
Annual Conference and Exposition, Salt Lake City, Utah, June 2004, 8 pages. 

V. Effect of Above Real Time Training and Post Flight Feedback in training of 
Novice Pilots in a PC-Based Flight Simulator by M. Javed Khan, Marcia 
Rossi, Bruce E. Heath, Syed Firasat Ali, Peter Crane, Tremaine Knighten, 
Christi Culpepper, Proceedings, Huntsville Simulation Conference, 
Huntsville, Alabama, October 2003, 10 pages. 

VI. Comparison of Different Methods of Grading a Level Turn Task on a Flight 
Simulator by Bruce E. Heath, Tomyka Crier, Proceedings, Huntsville 
Simulation Conference, Huntsville, Alabama, October 2003, 9 pages. 


3 


Blank Page 



Development and Assessment of a Novel Training Package 
For Basic Maneuvering Tasks on a Flight Simulator 

Using Self Instruction Methods and Above Real Time Training (ARTT) 

4. SUMMARY OF THE COMPLETE REPORT 

The report includes five research papers published in proceedings of various 
conferences and a Master of Science thesis. The research performed for the publications 
comprised three sets of experiments on training of novice pilots in a PC based flight 
simulator. The first set of experiments examined the use of Post-Flight Feedback (PFFB) 
and Above Real Time Training (ARTT) while training novice pilots to perform a 
coordinated level turn. The second and third sets of experiments investigated the effects 
of using out-of-the-window visual cues for training novice pilots to perform a basic flight 
maneuver. The flight maneuvers in the second and third sets were the straight-in landing 
approach and a coordinated level turn respectively. The out-of-the window visual cues in 
both sets consisted of visual hoops on the flight path through which a trainee would fly if 
maintaining the proper parameters. 

In all three sets of experiments, we evaluated a training strategy by requiring 
every participant trained within an assigned strategy, to fly a similar but more challenging 
task without the use of Above Real Time Training or feedback or visual cues. To 
authenticate the computed performance measures for the training and evaluation flights, 
we compared them with the grades awarded by certified flight instructors on a prescribed 
number of selected flights. 

For a comprehensive presentation of the training strategies covered in this 
research, a relatively small part of this report includes the work that is already reported on 
a preceding project performed by the same faculty members and sponsored by a NASA 
Dryden grant, NAG4-203 (At some places, by mistake, it was recorded as NAG4-209). 
Such material has sighted the appropriate reference in the enclosed publications. 

The following paragraphs identify the six publications and provide the respective 
abstracts. The abstracts in these paragraphs are abridged versions of the abstracts and 
conclusions or results in the respective publications. 


I. “An Evaluation of Training Interventions and Computed Scoring Techniques 
for Grading a Level Turn Task and a Straight in Landing Approach on a PC- 
Based Flight Simulator”, MS thesis of Bruce E. Heath, Major Professor M. 
Javed Khan, at Tuskegee University, May 2007, 76 pages. 

One result of the relatively recent advances in computing technology has been the 
decreasing cost of computers and increasing computational power. This has allowed high 
fidelity airplane simulations to be run on personal computers (PC). Thus, simulators are 
now used routinely by pilots to substitute real flight hours for simulated flight hours for 
training for an aircraft type rating thereby reducing the cost of flight training. However, 
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FAA regulations require that such substitution training must be supervised by Certified 
Flight Instructors (CFI). If the CFI presence could be reduced or eliminated for certain 
tasks this would mean a further cost savings to the pilot. This would require that the 
flight simulator have a certain level of ‘intelligence’ in order to provide feedback on pilot 
performance similar to that of a CFI. The ‘intelligent’ flight simulator would have at 
least the capability to use data gathered from the flight to create a measure for the 
performance of the student pilot. Also, to fully utilize the advances in computational 
power, the simulator would be capable of interacting with the student pilot using the best 
possible training interventions. 

This thesis reports on the two studies conducted at Tuskegee University 
investigating the effects of interventions on the learning of two flight maneuvers on a 
flight simulator and the robustness and accuracy of calculated performance indices as 
compared to CFI evaluations of performance. The intent of these studies is to take a step 
in the direction of creating an ‘intelligent’ flight simulator. The first study deals with the 
comparisons of novice pilot performance trained at different levels of above real-time to 
execute a level S-tum. The second study examined the effect of out-of-the-window 
(OTW) visual cues in the fonn of hoops on the perfonnance of novice pilots learning to 
fly a landing approach on the flight simulator. The reliability/robustness of the computed 
performance metrics was assessed by comparing them with the evaluations of the 
selected number of the level turn flights by two CFIs and the landing flights by three 
CFIs. 


Real Time Training (RT) with the post flight feedback resulted in the best 
perfonnance on the evaluation flights for a level turn as compared to no feedback or 
training with ARTT. Above Real Time Training (ARTT) using a sequence of 1 .5/2.0 with 
post flight feedback was observed to be the next best strategy for a level turn. The 1 .5/2.0 
ARTT implies that a trainee perfonned the first few flights with simulated events at a 
pace 1.5 times that of the actual, then flew with simulated pace 2 times that of the actual. 
The particular post flight feedback provided to the participants flying the level turn, 
consisting primarily of graphical feedback which showed the actual ground track 
superimposed over the target ground track, may be an effective fonn of feedback to 
implement in an "intelligent" flight simulator providing automated feedback. 

For the landing task, no statistically significant difference was found between the 
group provided with visual out-of-the-window cues of hoops and the control group, 
perhaps because visual cues in the fonn of the runway and visual approach slope 
indicator lights were already available for this maneuver. 

Poor correlations were observed between the evaluations by three different CFIs; 
improved correlations were, however, obtained for such flights on which the CFIs agreed 
on the best and worst controlled parameters. On those flights, the instructors’ grades 
correlated well with the computed grades. 


II. “An Experimental Study of the Effect of Out-of-the-Window Cues on 

Training of Novice Pilots on a Flight Simulator” by M. Javed Khan, Marcia 
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Rossi, Bruce Heath, S. Firasat Ali, Marcus Ward, Proceedings, Human 
Factors and Ergonomics Society Conference, San Francisco, California, 
October 2006, 5 pages. 

The effects of out-of-the- window cues on learning a straight-in landing approach 
and a level 360° turn by novice pilots on a flight simulator have been investigated. The 
treatments consisted of training with and without visual cues as well as density of visual 
cues. The performance of the participants was then evaluated through similar but more 
challenging tasks. It was observed that the participants in the landing study who trained 
with visual cues performed more poorly than those who trained without the cues. 
However, the performance of those who trained with a faded-cues sequence performed 
slightly better than those who trained without visual cues. In the level turn study it was 
observed that those who trained with the visual cues performed better than those who 
trained without visual cues. The study also showed that those participants who trained 
with a lower density of cues performed better than those who trained with a higher 
density of visual cues. 


III. “Con-elating Automated and Flight Instructor Assessments of Straight-In 
Landing Approaches by Novice Pilots on a Flight Simulator” by Bruce E. 
Heath, M. Javed Khan, Marcia Rossi, Syed Firasat Ali, Proceedings, 
Huntsville Simulation Conference, Huntsville, Alabama, October 2005, 7 
pages. 

The research reported in this paper was conducted to evaluate various performance 
metrics of a straight-in landing approach by 33 novice pilots flying a light single engine 
aircraft simulation. These metrics were compared to assessments of these flights by three 
flight instructors to establish a correlation between the two techniques in an attempt to 
determine a composite performance metric for this flight maneuver. The performance 
measures investigated were the standard deviation of the errors in runway alignment, rate 
of descent, airspeed and linear combinations of these deviations. Pearson Product 
Moment Correlations were calculated between these measures and the three individual 
instructors’ scores. The correlations ranged between 0.05 and 0.27. Correlations between 
the pairs of instructors’ grades were calculated and found to be moderate (r 2 = 0.4334, 
0.5054, 0.4334). Seven of the 33 flights were identified for which all three instructors 
agreed on both best and worst controlled parameters. Correlations between the instructor 
grades and the composite error metric (a linear combination of the standard deviations of 
errors in runway alignment, rate of descent and airspeed) were calculated for these 
flights. Strong correlations were found for all three instructors (r 2 ranged between 0.63 
and 0.82.) A composite error metric was used to provide the computed grades on all the 
training and evaluation flights. 
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IV. Using a Low Cost Flight Simulation Environment for Interdisciplinary 

Education by M. Javed Khan, Marcia Rossi, Syed F. Ali, Proceedings, ASEE 
Annual Conference and Exposition, Salt Lake City, Utah, June 2004, 6 pages. 

A multi-disciplinary and inter-disciplinary education is gaining increasing 
recognition as an essential element in the undergraduate engineering education. However, 
often the focus is on interaction between engineering disciplines. This paper discusses the 
experience at Tuskegee University in providing inter-disciplinary research experiences 
for undergraduate students in both Aerospace Engineering and Psychology through the 
utilization of a low cost flight simulation environment. The enviromnent, which is PC- 
based, runs a low-cost of-the-shelf software and is configured for multiple out-of-the- 
window views and a synthetic heads down display with joystick, rudder and throttle 
controls. While the environment is used to investigate and evaluate various strategies for 
training novice pilots, students were involved to provide them with experience in 
conducting such interdisciplinary research. On the global inter-disciplinary level these 
experiences included developing experimental designs and research protocols, 
consideration of human participant ethical issues, and planning and executing the 
research studies. 

An interdisciplinary team of aerospace engineering and psychology majors was 
involved in the interdisciplinary research project. The research provided the students an 
opportunity to function as a closely-knit team. They gained experience in: (a) 
Participation and interaction as a research group, (b) establishing tasks for achieving 
research objectives, (c) time and effort management, (c) interdisciplinary academic areas, 
(d) literature searches, and (e) formal presentations of results. The team successfully met 
the objectives set forth by the faculty mentors. The main elements of success were a high 
level of motivation of the students due to the opportunity of learning of concepts not 
covered in their major areas, students’ appreciation of responsibility of tasks with 
practical impact on the investigation and planning, and students’ involvement in actual 
conduct of the experiments under minimal supervision. However, it is essential that for 
interdisciplinary student research teams, the faculty mentors have a well-structured 
research plan to minimize overwhelming of the students. Regular meetings to provide 
well-defined tasks are an important element of the structure. 


V. Effect of Above Real Time Training and Post Flight Feedback in training of 

Novice Pilots in a PC-Based Flight Simulator by M. Javed Khan, Marcia 
Rossi, Bruce E. Heath, Syed Firasat Ali, Peter Crane, Tremaine Knighten, 
Christi Culpepper, Proceedings, Huntsville Simulation Conference, 
Huntsville, Alabama, October 2003, 8 pages. 

We investigated the use of Post-Flight Feedback (PFFB) and Above Real-Time 
Training (ARTT) while training novice pilots to perform a coordinated level turn on a 
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PC-based flight simulator. Participants in six different groups obtained their training in 
the following respective sequences: 

Group 2. 0/1. 5 ARTT no PFFB: Five missions in 2.0 ARTT then five missions in 1.5 
ARTT with no PFFB. (2.0 ARTT and 1.5 ARTT imply that the simulated events in the 
training missions were performed respectively at the pace 2.0 times and 1.5 times the 
actual pace of events) 

Group 2. 0/1. 5 ARTT with PFFB: Five missions in 2.0 ARTT then five missions in 1.5 
ARTT with PFFB 

Group 1. 5/2.0 ARTT no PFFB: Five missions in 1.5 ARTT then five missions in 2.0 
ARTT with no PFFB 

Group 1. 5/2.0 ARTT with PFFB: Five missions in 1.5 ARTT then five missions in 2.0 
ARTT with PFFB 

Group 1.0/1 .0 RTT no PFFB (Control group): Ten missions in RTT then with no PFFB 
Group 1. 0/1.0 RTT with PFFB: Ten missions in RTT with PFFB 

Immediately after the training missions/every participant conducted two challenging 
evaluation missions in real time with no feedback. The performance measure was based 
on the root mean squares of deviations in bank angle and altitude from their prescribed 
values. 

The study suggested that for training of a coordinated level turn: 

1 . If ARTT is to be used: 

(a) That it should be used as top-off training, i.e., after the pilot has 
experienced sufficient improvement in performance to have reached a 
plateau. 

(b) That it should be used in conjunction with appropriate PFFB. 

2. PFFB in a RTT is the most efficient combination 


VI. Comparison of Different Methods of Grading a Level Turn Task on a Flight 

Simulator by Bruce E. Heath, Tomyka Crier, Proceedings, Huntsville 
Simulation Conference, Huntsville, Alabama, October 2003, 7 pages. 

With the advancements in the computing power of personal computers, PC-based 
flight simulators and trainers have opened new avenues in the training of airplane pilots. 
It would be desirable to have the flight simulator make a quantitative evaluation of the 
progress of a pilot’s training thereby reducing the physical requirement of the flight 
instructor who must, in turn, watch every flight. 

In an experiment, University students conducted six different flights, each 
consisting of two level turns. The flights were three minutes in duration. Four kinds of 
grading have been used and compared for evaluating the flights. They are the following: 
(1) Instructor’s grade that is awarded by a certified instructor. (2) Automated grade that is 
based on prescribed tolerances in bank angle, altitude and airspeed monitored at several 
instants of time during a flight. (3) The performance index (P.I.) that is created by using 
the square root of the sum of the squares of dimensionless deviations in altitude and bank 
angle. (4) Perfonnance grade that is defined as 1 / (0.25 + P.I.) and provides a reasonable 
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4.0 grading scale. The comparison of performance grades with the other three 
performance measures suggests that they deserve further study for the evaluation of level 
turns in a simulator. 
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ABSTRACT 

AN EVALUATION OF TRAINING INTERVENTIONS AND COMPUTED SCORING 
TECHNIQUES ON A LEVEL TURN TASK AND A STRAIGHT IN LANDING 
APPROACH ON A PC-BASED FLIGHT SIMULATOR 


By 

Bruce E. Heath 

One result of the relatively recent advances in computing technology has been the 
decreasing cost of computers and increasing computational power. This has allowed high 
fidelity airplane simulations to be run on personal computers (PC). Thus, simulators are now 
used routinely by pilots to substitute real flight hours for simulated flight hours for training 
for an aircraft type rating thereby reducing the cost of flight training. However, FAA 
regulations require that such substitution training must be supervised by Certified Flight 
Instructors (CFI). If the CFI presence could be reduced or eliminated for certain tasks this 
would mean a further cost savings to the pilot. This would require that the flight simulator 
have a certain level of ‘intelligence’ in order to provide feedback on pilot performance 
similar to that of a CFI. The ‘intelligent’ flight simulator would have at least the capability to 
use data gathered from the flight to create a measure for the performance of the student pilot. 
Also, to fully utilize the advances in computational power, the simulator would be capable of 
interacting with the student pilot using the best possible training interventions. 

This thesis reports on the two studies conducted at Tuskegee University investigating 
the effects of interventions on the learning of two flight maneuvers on a flight simulator and 
the robustness and accuracy of calculated performance indices as compared to CFI 
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evaluations of performance. The intent of these studies is to take a step in the direction of 
creating an ‘intelligent' flight simulator. The first study deals with the comparisons of novice 
pilot performance trained at different levels of above real-time to execute a level S-turn. The 
second study examined the effect of out-of-the-window (OTW) visual cues in the form of 
hoops on the performance of novice pilots learning to fly a landing approach on the flight 
simulator. The reliability/robustness of the computed performance metrics was assessed by 
comparing them with the evaluations of the landing approach maneuver by a number of 


CFIs. 
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CHAPTER I 
INTRODUCTION 

Motivation 

One result of the relatively recent advances in computing technology has been the 
decreasing cost of computers yet increasing computational power. This has allowed high 
fidelity airplane simulations to be run on personal computers (PC). Thus, simulators are now 
used routinely by pilots to substitute real flight hours for simulated flight hours for training 
for an aircraft type-rating thereby reducing the cost of flight training. However, FAA 
regulations 1 (quoted below) require that such substitution training must be supervised by 
Certified Flight Instructors (CFI). 

5. AUTHORIZED USE. 

a. Instruction by an Authorized Instructor. Qualified PCATD's may be highly beneficial when 
used under the guidance of an authorized instructor to achieve learning in certain procedural tasks 
such as area departures and arrivals, navigational aid tracking, holding pattern entries, instrument 
approaches, and missed approach procedures. Accordingly, the FAA has determined to continue 
the policy that any time instruction is to be used to log time toward meeting any requirement of the 
regulations, an authorized instructor must have presented the instruction. 

If the CFI presence could be reduced or eliminated for certain tasks, this would mean 
a further cost savings to the pilot. This would require that the flight simulator have a certain 
level of ‘intelligence’ in order to provide expert feedback on pilot performance similar to that 
of a CFI. The ‘intelligent flight simulator’ (IFS) would have at least the capability to use 
data gathered from the flight to create a measure for the performance of the student pilot and 
provide expert feedback to the student pilot based on the computed performance index. Also, 
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to fully utilize the advances in computational power, the simulator would be capable of 
interacting with the student pilot using the best possible training interventions. 

Ludwig 2 et al. have referred to MulgandV main components of an Intelligent 
Tutoring System (ITS) to be the ‘helper’ and the ‘advisor’. In their implementation of an 
IFS, the ‘helper’ assisted the pilot by making the helicopter, at first, easier to fly. As the 
student became more comfortable with the aircraft, the pilot received less help. The 
‘advisor’ communicated directly with the pilot using text-to-speech software in four roles 
namely, tutorial, performance monitoring, monitoring flight control manipulation and 
advisory, verbalizing suggestions to control or correct flight. 

Objective 

The objective, however, of this study has been to investigate two important aspects of 
an IFS. These aspects being: (a) novel techniques to determine their efficacy in expediting 
the training of novice pilots and (b) reliability and robustness of computed performance 
indices in comparison to the CFI grading. 

The two novel training techniques that were evaluated during the course of this 
investigation were: 

(a) Above real time training (ARTT) for training a level S-tum with and without 
performance feedback. 

(b) Out-of- window (OTW) visual cues based on “tunnel in the sky” for training a 
landing approach. Performance feedback was used. 
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Literature Review 
Above Real Time Training (ARTT) 

In ‘above real time’ (ART), or as it has been referred to in literature 4 as ‘time 
compression’, events are presented in a simulator as if they were happening in real time but 
are actually running faster than the same event in the real world. For example, an aircraft 
flying at 90 knots might look as though it is flying at 180 knots. However, any airplane 
flying at twice the speed would have, for a level-turn, a 4 times larger radius. In fact, lift and 
drag for the airplane at 180 knots would be different from an airplane flying at 90 knots. 
ART, therefore, means that in a simulator, the aircraft flying at 90 knots will have lift, drag 
and airspeed indications of an aircraft flying at 90 knots, except that time will pass at a 
multiple of actual clock time. Thus, for example, at 2.0 ART, the airplane flying at 90 knots 
will cover 90 nautical miles in one hour of computer clock time, but in real clock time it will 
take 30 minutes. Previous experiments 4 ’ 5 ' 6 on training of pilots have indicated that real time 
training reinforced with ARTT could offer an effective training strategy for tasks which 
require significant effort at time and workload management. Kolf 5 documented that no 
matter how much pre-flight simulator training the pilot had undergone, the actual flight 
appeared, to the pilot, to take place in a much faster time frame. He chose pilots with 
experience flying the M2-F3 to fly experimental flights in the simulator and noted that by 
increasing simulator clock speed to 1.5 ART, the pilots felt that the simulator more closely 
resembled their actual flying experience. Kolf hypothesized that for ARTT an appropriate 
above real time factor would have to be chosen and would be a function of aircraft type, 
individual, task, and experience. Hoey 4 compared biomedical measurement data of test 
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pilots flying remotely piloted vehicles, with data taken in past flights. He suggested that 
providing ARTT on a simulator could approximately simulate the mental state of pilots as if 
actually flying. His suggestion was that ARTT compared with real time training on a 
simulator, would lead to a smoother transition from simulator to airplane. Crane and 
Guckenberger reference Schneider’s air traffic control study who proposed that “the 
primary effect of time compression is to allow more training trials within a given period of 
clock-time.” Crane and Guckenberger 5 contrasted this study by giving two groups the same 
number of trials so that the above real-time participants receive less clock-time than the real- 
time participants while performing the task of tracking and shooting at a target in the air. 
The test trials were then performed in real-time. The results of that study showed that 
participants trained in above real-time performed better than participants trained in real-time. 

However, this review of literature on ARTT determined that ARTT has not been 
exploited for the training of flight maneuvers. 

Tunnel in the sky 

The second training intervention investigated was the use of visual cues. This method 
is based on the concept of Tunnel in the Sky. The tunnel in the sky (Figure 1) which is a 
visual aid that shows the proposed path of flight for a maneuver in the sky is the subject of a 
number of studies, ' The benefits of using tunnel in the sky for navigation versus flight path 
tracking with conventional instruments have been investigated in these studies. 

Barrows, Alter, Enge, Parkinson, and Powell 10 compared the use of a tunnel display 
versus conventional instruments. In the simulator three different displays were offered: 
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1 . The control group: conventional instruments, vertical speed indicator (non-instantaneous), 
horizontal situation indicator (HSI) with glide slope display, attitude indicator, altimeter, 
airspeed indicator, and turn coordinator, 

2. First experimental group: Conventional instruments with a track symbol added to the HSI. 



Figure 1 . An example of tunnel in the sky. 

3. Second experimental group: The HSI from the first experimental group and an additional 
symbol “glide slope predictor”. 

4. Third experimental display: a pathway-in-the-sky. 

Using a simulator and actual flight, they found that the tunnel display offered significant 
improvements in horizontal and vertical flight precision and in workload reduction as 
compared to conventional instrumentation. 
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Doherty and Wickens 9 looked at the effects of preview, prediction, frame of reference 
and display gain in tunnel in the sky displays. In short, preview allows a pilot to anticipate 
upcoming demands and prepare a response before actually starting a maneuver. They also 
used a flight predictor in their display that reduced the cognitive effort required by the pilot 
in determining future trajectory of the airplane. An egocentric frame of reference was chosen 
to “preserve the visual relationships for tracking performance.” Their conclusions stated in 
part that preview and frame of reference (immersed viewpoint) provide the largest 
contribution to the tunnel benefit for flight path tracking. Prediction has a much smaller 
contribution. Mulder’s two studies ’ discuss the effect of the optical information conveyed 
by a pathway-in-the-sky as it relates to straight and curved trajectories. The pathway was 
drawn using a rectangular tunnel. Mulder credits the worth of the tunnels to texture gradients 
that provide the useful information about the participants’ motion. In Mulder’s study, 
parallel lines to the viewing direction convey optical “splay angle” information or gradient of 
perspective (lines that meet at infinity). Lines that are perpendicular to the viewing direction 
convey an optical density or a “gradient of compression” (Figure 2). The study found that 
for a straight tunnel, optical splay and optical density were “essential for the pilot to perceive 
and control the aircraft position and motion with respect to the tunnel.” Also, Mulder found 
that splay and density were complementary. A tunnel display without optical splay lines is 
not recommended and a tunnel display with the wrong number of frames could cause clutter. 

For a curved trajectory, gradients have a much smaller use. First of all there is no 
vanishing point as can be seen in a straight tunnel. Second, only the gradients conveyed by 
the nearby elements of the tunnel provide valuable information about the aircraft movement. 
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Third, the presented curved trajectory does not convey actual position and attitude relative to 
the trajectory. These are presentation biases that lead the pilot to compensate for position 
and attitude errors that are not really there. 



Figure 2. Straight tunnel components. Q shows the optical splay angle, s,v,rj,fj. show the 
perpendicular lines that convey optical density (Mulder J 1 ). 

For a curved trajectory it was found that splay lines are substantially but not 
significantly useful. Presentation bias is greatest when the display shows only contour lines 
and decreases considerably when tunnel frames are used. Presentation bias leads to errors in 
the lateral position relative to the center circle. Participants found it difficult to tell when 
they were flying on the center of the path. Also, Mulder recommends use of a greater 
number of rings to reduce presentation bias. 
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Performance Feedback 

As mentioned above, the cost component associated with the requirement of a CFI to 
be present during every simulation can be minimized if the flight simulator is able to emulate 
expert feedback of a CFI. Vruels and Obermayer 13 emphasize the need of automated 
performance measures on modem simulators as a substitute for direct evaluation of 
performance by an instructor. Rantanen and Talleur 14 have suggested the following five 
primary measures for pilot perfonnance from the flight data recorder for nine flight 
parameters: (1) Standard deviation. A small standard deviation is indicative of good 
performance. (2) Root mean square error (RMSE) which summarizes the overall error. (3) 
Number of deviations, which is a measure of occurrences of the aircraft staying outside the 
prescribed tolerances. (4) Time outside tolerance is the cumulative time the aircraft spends 
outside prescribed tolerances. (5) Mean time to exceed tolerance is computed from the rate 
of change between successive data points and the aircraft's position relative to a given 
tolerance. Heath and Crier 11 used computerized scoring and certified flight instructors (CFI) 
to attempt to devise a computed perfonnance measure for a level turn. A performance index 
based on deviations from the required heading, altitude and airspeed was compared with the 
instructors’ letter grades. One of the challenges of such comparisons is inter-rater reliability 
of the flight instructors. In a study about crew resource management (CRM) in a simulated 
helicopter, Brannick, Prince, and Salas 15 evaluated three items: “(a) specific crew behaviors 
in response to scenario events (e.g., whether the crew kept out of icing conditions), (b) 
valuations of crew responses to scenario events (e.g., overall handling of the icing problem), 
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and (c) crew resource management (CRM) dimensions for the entire scenario (e.g., 
evaluations of decision making).” 

They were able to show that “respectable levels of both interjudge agreement and 
internal consistency were achieved for items dealing with the evaluation of crew performance 
in response to items in events embedded in the scenario”. Interjudge agreement was low but 
internal consistency was high on CRM items and scales. Interjudge agreement was high but 
internal consistency low for specific observable behaviors. 

An attempt to correlate instructors’ grades with a performance metric by Heath and 
Crier 16 showed the same result. The instructors did not agree on how to measure 
performance. Discrepancies between instructors may be due partially to the fact that 
instructors normally give a pass/fail type of evaluation and not the type of A, B, C, D, E 
grade requested for the current study. 

In a follow-up study the instructors were asked to identify the parameter they thought 
the student best controlled and the parameter they thought was worst controlled. 
Performance metrics were graphed against the grades given by the three instructors for the 
flights where they were in agreement of the best and worst controlled parameters. Using that 
kind of selection, a better correlation was found between instructor grades. 
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CHAPTER II 

LEVEL TURN EXPERIMENT 

This chapter describes a study which compared Above Real Time Training (ARTT) 
versus Real Time Training to investigate the performance of 32 novice trainees using a Level 
Turn Maneuver. The turn flight maneuver is one of the basic flight maneuvers required for 
a pilot in pre-solo training. According to the Federal Aviation Regulations other basic flight 
maneuvers include takeoff and landing, straight and level flight, climb and descent. (FAR 
2002; §61.87) The purpose of this experiment is to investigate the effect of different levels 
of ARTT on novice pilots and to attempt to create a performance metric that will allow the 
comparison of the various groups of students flying at different levels of ART. 


Setup 

Hardware 



Figure 3. LiteFlite partial mock cockpit setup shows heads down display and OTW. 
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The experiment was performed in the Flight Vehicle Design Lab at Tuskegee 
University. The “simulator” consists of a mock setup of a partial construction of a cockpit. 
Three 1 9” monitors create a panoramic, outside of the window (OTW) view. The center 
monitor contains a heads up display (HUD) superimposed on the forward view. The HUD 
displays a composite of several relevant instruments including: altimeter, airspeed indicator, 
horizontal situation indicator (HSI), turn and slip coordinator, and heading indicator as 
shown (Figure 4). 



Figure 4. HUD for the LiteFIite software is superimposed on the center monitor of the OTW view. 


The mock setup had an inside the cockpit “instrument panel” view displayed on a 15” 
monitor as shown (Figure 5). 

The joystick used was a Saitek X36F and the throttle was a Saitek X35T controller. 
Rudder pedals used were from CH products. The PCs for the simulator were two Heavy 
Metal computers made by Quantum 3D. 




Artificial Horizon 


'''(As' 1 ' 
Indicator 


0 Heading 


Figure 5. LiteFlite Fiead down display (HDD). 


Each computer has 2 Pentium II processors running at 400Mhz, 400MB RAM, and 


has three extra video cards for the Out of the Window (OTW) view. Figure 3 also shows a 


moving map display monitor on the right of the picture. The moving map display was not 
used in this experiment. Additional details can be found in Williams 17 . 


Software 

Flights were conducted using the LiteFlite version 3.3 flight simulation software 
developed by SDS International, Orlando, Florida. LiteFlite offers the Predator (UAY) 
simulation which was used in these studies. LiteFlite has an internal facility to capture flight 
data using Visual Basic. The same feature exports the data to *.csv files that are readable by 
Microsoft’s Excel spreadsheet program. 
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Method 

Participants 

The student participants for this study were 36 undergraduate freshmen at Tuskegee 
University who had little or no prior experience with an airplane simulator. All participants 
enrolled in an Introductory Psychology class. The student participants were offered extra 
course credit in their psychology course for their participation. Our acceptance of a 
participant into the experiment, beyond the familiarization phase, was based upon their 
adherence to the flight parameters described in Table 1 . Participants who achieved a score of 
2.0 (grade based on comparing parameter values in the flight with the parameter values and 
tolerances required) on at least one familiarization flight were allowed to continue into 
training. The grading scheme is explained in Table 1 . Thirty two participants who achieved 
the passing grade continued on to the training phase. Each of the participants who achieved a 
score of 2.0 was randomly placed in one of six groups. 


Table 1 . Grading Criteria for Straight and Level Flight 


Nominal Altitude = 10000 ft 

Nominal Speed = 129 knots 

Nominal Bank angle 
= 0 degrees 

Grade 

Deviation(A in altitude) 

Deviation(Ain speed) 

Deviation(A in degrees) 



A = ± 50 

A = ± 5 

A = ± 3 

A = 

= 4.0 

+50 < A < +100 

+5 < A <+10 

+3< A < +6 

B = 

= 3.0 

-100 < A < -50 

-10 < A < -5 

-6 < A < -3 



+100 < A < +150 

±10<A<±15 

+6< A < +9 

C = 

= 2.0 

-150 < A <-100 

-15 < A <-10 

-9 < A < -6 



+150 < A <+200 

±15 < A <±20 

+9 < A < +12 

D = 

= 1.0 

-200< A <-150 

-20 < A < -15 

-12 < A < -9 



+200 < A or A < -200 

+20 < A or A < -20 

+12 < A or A < -12 

E = 

= 0.0 
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Experimental Design 

This experiment used a 3X2 factorial design. There were three types of training used: 
all flights flown in real time (RT), the first half of the flights flown in 1.5 above real time and 
the second half flown at 2.0 above real time (ART), or the first half of the flights flown at 2.0 
ART and the second half of the flights flown at 1 .5 ART. Half of the participants in each 
group were randomly assigned to receive feedback or no-feedback during training. Feedback 
for this study took the form of the student instructors making suggestions to the participant as 
to how to improve the handling of the airplane. For the feedback group, students were given 
a picture representation of their performance in the form of a ground track on the second, 
fourth, sixth, eighth, and tenth flights. For the study conducted on the flight simulator, all 
participants experienced the following segments: Orientation to Simulator Controls and 
Functions, 2 Demonstration Flights, 3 Familiarization Flights, 8 Training Flights, and 2 
Evaluation Flights. The evaluation flights were all conducted in RT. 

Procedure 

All participant pilots experienced the following sessions: Orientation to flight controls 
and aircraft instruments, familiarization flights, training flights and evaluation flights. 

The instructors individually attending the participants in this experiment were the 
author, from the Aerospace Science Engineering Department, and three other juniors and 
seniors in the Psychology Department at Tuskegee University. The undergraduate instructors 
had, for the most part, no flying or simulator experience so they spent a good deal of time 
becoming comfortable with the controls and functions of the simulator. The graduate 
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instructor has 10 hours of flight time in a Cessna 172 and numerous hours using Microsoft 
Flight Simulator. 

Orientation to Flight Controls and Aircraft Instruments 
A protocol was formulated to introduce the novice subjects to the flight controls. 
Such a protocol was considered essential so as to ensure standardized briefing to all 
participants thereby minimizing the influence of different instructors. During the orientation 
process, participants were shown a model airplane and informed of the basic control surfaces 
of the aircraft, their functions, and the movements associated with each control surface. The 
participants were then given an overview of the locations and functions of the joystick, 
rudder pedals, and the throttle located in the mock cockpit. Then, participants were 
instructed in the location and functions of the following displays on the HUD (head up 
display): altimeter, radio altimeter, airspeed indicator, heading tape, artificial horizon, pitch 
ladder, and clock. 



Figure 6. LiteFlite HUD and HDD. 

Participants were also instructed on the location and functions of the following 
instruments on the HDD (head down display): artificial horizon and the vertical velocity 
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indicator. Because the tum-and-slip indicator on the HDD was not functioning properly, 
participants were explained the use of the HUD to determine if the aircraft was making a 
coordinated turn. 

Familiarization Flights 

Participants observed the experimenter flying one 3-minute straight and level flight 
while explaining the controls and operation of the aircraft. Participants flew four flights of 3- 
minutes duration each in which they were told to maintain airspeed of 129 knots, a heading 
of 360 degrees, and an altitude of 10,000 feet. The participants were coached on all but the 
last of the flights. On the first flight, the experimenter asked the student to focus on altitude 
maintenance primarily. On the second flight the experimenter asked the student to focus on 
altitude and heading. For the third flight the experimenter asked the student to maintain 
altitude, heading and speed. The experimenter assisted with the controls, if needed. At the 
end of each of the flights participants were shown their actual course and given feedback on 
how well they performed. On the last flight, the participant was to maintain altitude, heading 
and airspeed without being coached. 

During the familiarization flights, participants were graded on a 4.0 grade scale. The 
grades were calculated by a program written in visual basic by SDS. 

Training Flights 

The experiment consisted of a training phase and an evaluation phase. In the training 
phase pilots flew a mission consisting of a coordinated 1 80° turn with a 1 0° bank angle while 
the evaluation task was an S-turn with a bank angle of 30°. Each group conducted their 
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training in the following sequence immediately followed by two evaluation flights in real 
time. Participants in Groups II, IV and VI were given post-flight feedback (PFFB). 

Group I: Five missions in 1 .5 ARTT then five missions in 2.0 ARTT with No-PFFB, 
Group II: Five missions in 1.5 ARTT then five missions in 2.0 ARTT with PFFB, 

Group III: Five missions in RTT then five missions in RTT with No-PFFB, 

Group IV: Five missions in RTT then five missions in RTT with PFFB Feedback, 

Group V: Five missions in 2.0 ARTT then five missions in 1 .5 ARTT with No-PFFB, 
Group VI: Five missions in 2.0 ARTT then five missions in 1.5 ARTT with PFFB. 

Training Flights Briefing Scenario 

To provide some realism to the training, the participants were briefed on their task 
according to the following scenario: 

“You’re the flight leader of a combat air patrol on a routine mission at 5000 ft 
altitude, 129 knots and heading in a 360° direction. An unidentified low-speed aircraft is 
being tracked by radar moving towards a no fly zone. Despite repeated requests, the aircraft 
fails to identify itself and continues towards the no fly zone. The Air Defense Radar 
Controller assigns the interception to your flight in order to make a visual identification. You 
are asked to initiate a right hand turn, maintain a bank angle of 10 degrees, an altitude of 
5000 ft and a speed of 129 knots. Exactly after two minutes (as recorded on your heads up 
display), you should level out. Your heading should be 180°, putting you right behind the 
target, close enough for visual contact. You will complete a number of these missions. Do 
your best, because once you have finished the practice missions, you will be required to 
complete a test mission with a slightly more difficult task.” 
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Feedback in Training 

Participants in the No-PFFB groups received no feedback during or after flights. If 
they asked questions about the task, they were re-read the relevant instructions. Participants 
in the PFFB groups received verbal feedback after each flight. In verbal feedback, the 
experimenter told the participant the errors in manipulating the controls and in focusing on 
the wrong instruments. After the second, fourth, sixth, eighth, and tenth flights, participants 
in the PFFB groups were also shown a screen printout of the desired ground track and the 
pilot’s actual ground track for comparison. Then, the experimenter explained possible 
reasons for the deviations in performance. 

Evaluation Task 

The evaluation flight was an S-turn. Pilots were instructed to make a 30° banked 
right turn to 180° or south and then an immediate 30° banked turn to 0° or north. The 
participants were told that they would fly two more flights having a different and more 
challenging task. Both flights were conducted in real-time for all participants and lasted for 
three minutes each. No feedback or coaching was provided during testing. Participants were 
read the following scenario before each flight. The participants were to fly an S-Tum with 
30° bank angles, maintain 5,000 ft. and a speed of 164 knots. 

However, a loss of situational awareness was observed in most of the participants 
after the first leg of the S-tum during evaluation. Those people who lost situational 
awareness could be seen doing some of the following: continuing the turn in the same 
direction, banking the plane and then pulling the stick back to increase the rate of turn, or 
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continuing in a straight line after the first turn. Thus, only the first leg of the S-tum was 
evaluated for analysis purposes. 

Testing Scenario 

The participants were given the following scenario for evaluation flights: 

“As part of a Combat Air Patrol, the Air Defense Radar Controller notifies you that a 
hostile aircraft is being tracked heading towards a vulnerable point (VP). Your instructions 
are to initiate a right hand turn in order to pursue the aircraft while maintaining a bank angle 
of 30°, an altitude of 5000 ft., and a speed of 164 knots. Exactly 48 seconds after the 
instructions to initiate the turn you should level out. You will now be exactly behind the 
target on a heading of 180°. You are then informed of a second target. You should fire your 
missile and immediately initiate a left turn with a 30° bank maintaining 5,000 ft. altitude and 
a speed of 164 knots in pursuit of the second target assigned. Exactly 48 seconds after 
initiating the second turn you should level out. Your heading should be 360°, putting you 
right behind the second target. Launch your missile. Mission accomplished.” (It should be 
noted that no missile function was provided in the simulation). 

Performance Metrics 
Familiarization flight performance metric 

The performance metric for the level turn was a letter-grade scheme based essentially 
on grading scheme of Williams. 17 The same scheme was used for selecting participants for 
the study (Table 1 ). 
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Turn Task Performance Metrics 

The performance metric to track learning was calculated. This performance measure 
was based upon an aggregate of non-dimensional altitude error and the error, in radians, for 
the bank angle. The two measures were then squared, and the square root was taken of that 
result. Hence: 

Error = ^(AH / s 3?) 2 + A<f~ 

where the radius of turn ‘Y’ was calculated for the flight parameters to be maintained by the 
pilot using the following relation 16 : 

V 2 1 

Y = -—= ; n =— , g = acceleration due to gravity for the V = airspeed, 

g.hi 2 - 1 cos</> 

and: (f> = turn bank angle 

Here, altitude error (AH) is non-dimensionalized by the theoretical radius (Y) of the turn 
made by the simulated airplane. If H is not divided by Y small changes in altitude 
overwhelm any contributions by possible large changes in bank angle (<j>). Also, error in 
speed was not used in this calculation because the pilots did not make use of the throttle. 
Hence, as altitude decreased, speed increased and vice versa. The effect of speed changes 
was thus captured in the error in altitude as can be clearly seen in Figure 7. 




Figure 7. Deviations of altitude and speed (typical for a single flight). 

Results And Discussion 

The graph in Figure 8 shows the average errors for each group during training and 
evaluation flights. For the ART training regimen the first five flights and last five flights 
were at either 1 .5 above real time or 2.0 above real time. In either case after the fifth flight 
the one level of above real time was changed to the other. 
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Figure 8. Average errors during training and evaluation flights. 


In terms of post flight feedback or no post flight feedback, the graph shows that for 
both 1. 5/2.0 ART and 1. 0/1.0 RT, the No-PFFB group did worse than the corresponding 
PFFB group. This was expected because each participant in the PFFB group was given the 
advantage of knowing how well or poorly he was doing. For the 2.0/1 .5 group the No-PFFB 
group actually did better than the corresponding PFFB group. This second result was 
unexpected in that feedback should provide extra help for the participant to understand what 
is going wrong. This may be due to the fact that the PFFB started the experiment with a 
worse performance; thus perhaps there was a difference between the two groups from the 
start of the experiment. 

It might be expected that for the PFFB and No-PFFB groups, the performance of the 
1. 5/2.0 group should be similar to the 2.0/1. 5 group performances at the very beginning as no 
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advice was given to either group. For 1 .0/1.0 No-PFFB, this was not the case. At the start of 
training with no feedback given, average performance by flight number for the No-PFFB 
group was much worse than the PFFB group. This continued throughout the training phase. 

The transition between flights 5 and 6 for the 2.0/1 .5 flights showed that for one 
flight, the PFFB group did better than the No-PFFB group and then proceeded to do worse 
for the rest of the training. This could have been from a loss of confidence in what the 
instruments were showing the participant and in the instructions given from the instructor. 
For the No-PFFB group they only relied on the instruments and scripted instructions from the 
experimenter. 


CHAPTER III 


LANDING TASK EXPERIMENT 

This chapter presents an experiment in which the effect of Out-of-the Window 
(OTW) visual cues for training of novice pilots for a straight in landing approach on a flight 
simulator was studied. A performance metric based on RMS errors in the landing flight 
parameters was used. To validate the chosen computed metric, the computed grades for a 
few selected landing flights were compared with grades determined independently by three 
certified flight instructors by viewing the flight video of the approaches. 

Setup 

Hardware 

The Landing Study was performed in the Flight Vehicle Design Lab at Tuskegee 
University. The experimental setup consisted of one computer with 1 5” monitor used as the 
instrument panel or heads-down-display (HDD); three other computers with monitors were 
used for OTW. The physical setup, shown in Figure 9, is a mock setup of a partial cockpit. 

The three OTW view computers are all Gateway E series computers with the 
following specifications: Pentium 4 2.60GHz processor Hyper-’ Threading Technology with 
512KB cache, 512 MB ram, bus speed 800 MHz and memory speed of 333 MHz; NVIDIA’s 
GeForce FX5200 is the video card used for the three OTW view computers. The “master” 
computer, which includes the HDD, is a Dell with following specifications: Pentium 4 
2.4GHz processor with 512 KB cache, 512 MB ram, bus speed 133 MHz. The video card is 
a NVIDIA GeForce Ti 4600 with 128MB memory. A KVM (keyboard, video, and mouse) 
switch was used to provide keyboard and mouse service to all computers. Three 19” 
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monitors were used to provide the 135 degree out of the window (OTW) view Figure 9. The 
fourth monitor is a 15” monitor used to view the instrument panel. All of the four computers 
use 100 megabit Ethernet cards for LAN access. The computers are physically connected 
using a 5-port 100 megabit switch. CH rudder pedals and Thrustmaster Cougar HOTAS stick 
and throttle were used for flight and engine controls. 



Figure 9. Partial mock cockpit. MSFS2002 shows the hub (switch) OTW view, HDD, switch and Thrustmaster 
joystick and throttle (partially hidden) 

Software 

The operating system for each computer is Microsoft Windows XP. Microsoft Flight 
Simulator (MSFS) 2002 was installed in the default location on each PC. In order to create 

17 . 

the panoramic OTW view, Wideview’s *.dll file was installed. Wideview is a program that 
will allow multiple instances of Flight Simulator to be controlled by a “master” computer so 
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that a panoramic view can be attained. Using Wideview, it would be conceivable to have a 
360° display, given enough computers. As a prerequisite for Wideview’s capability to 
synchronize OTW view displays, FSUIPC.dll and the IPX/SPX protocol (for Windows) were 
installed. FSUIPC. 18 is a pathway for programs like Wideview to talk to MSFS 2002. The 
flight recorder software module FLTREC.dll 19 was installed to record the data to a *.dat file. 
FLTREC’s output is configurable through a settings menu item. The data in the *.dat file is 
in XML format. A sample spread sheet was used to translate the fltrec.dat file into Microsoft 
Excel workbook columns for a spreadsheet. An additional worksheet was created to 
determine the performance (based on a 4.0 scale). The 4 point grade scale for straight and 
level flight was similar to that used by Williams .. The grade was based on values in certain 
ranges for parameters of level flight. 


Method 

Experimental Design 

This experiment consisted of a between groups design with the independent variable 
being the type of visual cues presented. The three levels of the variable were: all training 
flights with OTW cues, all training flights without OTW cues, and a few training flights with 
OTW cues while others without OTW cues. 

Participants 

The student pilots participating in the experiment were all freshmen attending 
Tuskegee University. As an incentive, extra credit in an Introductory Psychology course was 
awarded to every participant. The volunteers that were selected had little or no prior 
experience with flight simulators or piloting an airplane. Every volunteer, after filling in 
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consent and background forms (Appendix C), attended a brief orientation session and 

conducted five familiarization flights. The orientation session introduced the various 
controls and instruments that the participant would use during the experiment. Three of the 
familiarization flights required the student to fly straight and level. Any volunteer who 
scored a 3.0 or better on at least one of the familiarization flights was selected for the training 
and evaluation sessions. The grading matrix is given below in Table 2. The excel Visual 
Basic module that calculated the grades is shown in Appendix A. 

Throughout the experiment, every participant was individually attended by at least 
one of the three available experimenters. The experimenters were the author, and two other 
undergraduate senior Psychology students. None of the senior Psychology students had any 
experience on a flight simulator, so they spent a good deal of time becoming familiar with 
the equipment by asking questions and teaching each other the operating procedures. 

Procedure 

Orientation to Flight Controls and Aircraft Instruments 
For the Orientation session, pilots were shown the relevant instruments: airspeed 
indicator, vertical speed indicator, artificial horizon, gyroscopic compass and RPM gage 
shown in Figure 10. Appendix D includes the set of instructions that were given to every 
volunteer in the orientation session. The pilot controls introduced were: Joystick, throttle, 
and rudder pedals. Instructions were given on how to operate the controls and the 
consequences of operating them. For example, ‘pulling back on the joystick pitches the nose 


up’. 


Blank Page 
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Familiarization Flights 

Each participant flew 5 familiarization flights. The first three familiarization flights 
were straight and level flights of three minutes duration each. If necessary the experimenter 
could provide hands on help to steady the airplane in flight. During the straight and level 
familiarization flights, participants were graded on a 4.0 GPA scale. The scores were 
computed using a Visual Basic 6.0 code that compared the parameter values in flight 
recorded every second with pre-determined parameter values and tolerances. The pilot was 
required to fly 5,000 ft, 75 knots, 0 degree bank angle. For each flight a score was calculated 
based on the following criteria: 


Table 2. Grading Criteria for a Straight and Level Flight 


Nominal altitude = 5000 ft 

Nominal Speed = 75 knots 

Nominal Bank angle = 0 
degrees 


Deviation(Ain altitude) 
feet 

Deviation(Ain speed) 
knots 

Deviation(A in bank 
angle) degrees 

Grade 

o 

u-> 

+1 

II 

<1 

A = ± 5 

A = ± 3 

A 

+50 < A < +100 

+5 < A <+10 

+3< A < +6 

B 

-100 < A < -50 

-10 < A < -5 

-6 < A < -3 


+100 < A < +150 

±10 < A < ±15 

+6< A < +9 

C 

-150 < A <-100 

-15 < A < -10 

-9 < A < -6 


+1 50 < A < +200 

±15 < A <±20 

+9 < A < +12 

D 

-200< A <-150 

-20 < A <-15 

-12 < A < -9 


+200 < A or A < -200 

+20 < A or A < -20 

+12 <A or A<-12 

E 


Those participants achieving a score of 3.0 or greater continued with the training. 
The fourth and fifth flights familiarized the pilot with making a straight-in landing initiated 
approximately from an altitude of 2860 ft with 500 ft/min rate of descent without any OTW 
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cues. In both flights, the experimenter provided verbal feedback to the pilot to assist in 
making appropriate corrections. In the fifth flight, the experimenter gave instructions to the 
pilot to listen to the voice from the simulator and where necessary, make appropriate 
corrections. After each of the familiarization flights, a graphical record was shown to the 
pilot using Microsoft’s Flight Analysis Tool. This graphical record showed a trace of the 
altitude and heading of the aircraft. 

Training Flights 

The training session required every pilot to fly eight straight-in landing flights from 
an altitude of approximately 2860 ft and at a heading of 192 degrees thus aligned with the 
centerline of the runway. The task given to the pilots was to maintain 75 knots airspeed, 
descend at 500 feet per minute (by reading both the vertical speed indicator and by looking at 
the V.A.S.I (Visual Approach Slope Indicator) lights on the left side of the runway, and head 
towards and line up with the centerline in a straight-in approach to the Bremerton 
International Airport in the Seattle, Washington area. Each pilot was asked to land on the 
centerline of the runway. The aircraft being flown was a Cessna 1 72. The flight scenario was 
a modified version of a landing lesson from the “flight school” portion of Microsoft Flight 
Simulator, MSFS 2002. Participants selected for training were randomly assigned to the 
three groups named HH, NH and FH. A trainee in group HF1 was required to conduct all of 
the eight training flights with hoops as OTW cues. The flights with the hoops were a 
modified version of the ‘Landing Lesson IF of the MSFS 2002. The hoops in this lesson are 
placed on the glide-slope. It was explained to the participants that if they maintained the 
required flight parameters, they would automatically pass through the hoops. For the NH 
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group a trainee conducted all of the flights without the OTW cues. A trainee in group FH 
was required to conduct eight flights some of the flights had hoops and the remaining flights 
had no hoops. The selected density was ‘5 hoops in sight at a time’, which was the lowest 
setting in Microsoft Simulator. 

Feedback in Training Flights 

After each training flight, participants were able to look at the flight analysis provided 
by Microsoft Flight Simulator 2002. In the analysis the student could see a quick playback 
of a trace of the flight and see whether the airplane was on the centerline of the runway and 
whether it was descending at a constant rate. MSFS2002 also provided a computerized voice 
for training a landing task. 

Evaluation Flights 

After the training flight two evaluation flights were flown. The same scenario as the 
training flights was used. The only difference was that all trainees had to fly the glide slope 
without hoops. The sound was turned off so that the trainee could not hear the computer 
voice. Also, Microsoft’s version of mild turbulence was added to make the task more 
challenging. No feedback was given after each of the flights. The total duration to complete 
the three phases for each participant was approximately 2-1/2 hours. 

Observation 

At the end of the experiment, it was noted that all participants in the FH group were 
given two distinctly different sequences of hoop flights. Therefore, the FH group was 
dropped from the study. However, the evaluation flights were used for the CFI study reported 
in Chapter IV. 
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Performance Metrics 

Metric Definition 

Data were gathered using third party flight recorder software Fltrec9.0 software. This 
software writes its data to an .xml file and was interpreted by a shareware spreadsheet into 
MSExcel for each flight. The analysis originally included data from the time the simulator 
was un-paused to the time the pilot should have touched down. This was not really accurate 
in that sometimes before the runway threshold, pilots must reduce power; change the airplane 
rate of descent (and airspeed) to make a landing. We decided to calculate the time it should 
have taken an airplane descending at 500 ft/min to hit the ground (which is what would 
happen) and compare this to the actual flight. Most of the flights exceeded this time. 

The following parameters were taken into consideration for determining the landing 
task performance metric: rate of descent, runway alignment, and airspeed. Thus error in 
speed is (AV = V r - V.f(t)), and error in rate of descent is: (AROD = H.f(t) - H r (t)) 
where H r = H sta rt - 8.333ft/sec * t 

The performance metric used assumed the airplane was on the extended centerline of 
the runway. For each second an airplane has a heading and speed. Then for each second, the 
airplane will have a distance Ax that it will move laterally from or toward the centerline. The 
distance the airplane travels, in a straight line, is: = V*t. 

Error in the heading, (Avj/) of the airplane is the difference between the runway heading and 


the current heading. 
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Figure 1 1. Runway Alignment. 


The straight-line distance and the At)/ heading, lead to the distance 
Ax(t) = V(t)*t*sin(Atj/(t-l)) 


Error in heading (A = \\i (t 0 ) - \\i( t 0 +l)), i|/(t) heading at time t is in radians 


Since velocity is in knots conversion to feet was necessary 
AY ft/s = 1 .687809 * (AV knots) 


Variables Latl through Lat5 (Figure 12) were Degrees from the Prime Meridian and 
converted to radians for calculation. Error in each of these parameters was determined for 
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every second of the flight by taking the difference between the required value of the 
parameter and the value of the parameter during the flight. 

V r is the required velocity Vf(t) is the velocity at time t 

v|/ r is the heading in radians i|/.f(t) is the heading at time t 

H. r is the required altitude Hf(t) is the altitude at a time t. 

Hstart is the starting altitude Ax is the lateral error from centerline 

H r (t) is the altitude at time t assuming a descent rate of 8.333 ft/s 

The errors, AV, Ax, and AROD are calculated for each second of a flight which could 
vary from 300 - 420 seconds. The RMS errors for AV, Ax, and AROD were computed for 
the entire flight. A linear combination of the RMS errors was used as the metric for 
evaluating the performance of the segment of the flight where the airplane should have 
constant airspeed, rate of descent and heading. 

Starting Error 

The airplane however, did not start out on the centerline. So each participant had to 
align the airplane and descend at the proper rate and the correct speed. This slight offset was 
barely discernable on the monitor but very noticeable in the data. To identify this problem, 
Latitude and Longitude data were gathered and used as “Cartesian coordinates.” The straight 
line distance between the current Lat/Long and the centerline Lat/Long would then be the Ax. 
Using this method it was determined that the flight started out about a third of a mile 1 600 ft 
left of the centerline. 

The method used to calculate the distance from the centerline Ax starts with the figure 
shown in Figure 12. The latitude and longitude for the point on the centerline was 
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determined with the method below. The straight line distance in kilometers from the 
centerline (Ax) was determined using equation 6 below 15 that comes from the Spherical Law 
of Cosines 15 and using 6371km as Earth’s radius (R). 
d = acos(sin(lati).sin(lat 2 )+cos(lati).cos(lat 2 .).cos(long 2 -longi)).R 



Figure 12. Runway alignment error geometry. 


Lat5 = .98 1 98 l(Long3) +168.043 (Eq. 3.1) 

Centerline coordinate distance = (Lat3 - Lat5)Cos(tan" 1 (.981981981)) ( Eq. 3.2) 

longitude_coordinate_distance = Centerline_coordinate_distance*Cos(tan' 1 (- 1 /.98 1981981)) 
- — ( Eq. 3.3) 

Long4 = Long3+longitude_coordinate_distance ( Eq. 3.4) 


Lat4 = .981 98 l(Long4)+l 68.043 


( Eq. 3.5) 
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Alignment Error 

Ax (km)^ ArcCos(sin(Lat3)*sin(Lat4) + Cos(Lat3)*Cos(Lat4)*Cos(Long3-Long4))* 6371 
(ref. 15)— (Eq. 3.6) 

Ax (ft) = Ax(km) *1000meter/km*3. 2808339 ft/meter ( Eq. 3.7) 

The flight data were processed to determine the error in airspeed, rate of descent and 
runway alignment. A linear combination of the non-dimensionalized standard deviations of 
errors in the three parameters was then calculated and used as a composite error metric for a 
flight as well. 


Results and Discussion 

The performance metric was plotted against flight number for the purpose of 
determining whether student pilot’s performance improved over time. In this performance 
metric, performance improvement means a decrease in the value of the metric. Figure 14 
presents the performance metric, normalized with the first training performance, for every 
participant in the No Hoop (NH) group and for every flight in the training and testing 
sessions. Likewise, Figure 16 presents the performance metric, normalized with the first 
training performance, for every participant in the HH group. Figure 17 presents the 
performance metric of every flight in the training and testing sessions for the NH and HH 
groups separately averaged over the 12 participants per group. A simultaneous study of 
Figures 14, 16, 17 leads to the following observations. 

In both NH and HH groups, most of the pilots have improved performance in the 
second training flight (T2) when compared with the first training flight (Tl). In any typical 


48 


training program it is expected that the performance of trainees would show a visible 
improvement with the amount of training; surprisingly, such a trend is not seen here for the 
six training flights in sequence from T3 to T8. For the HH group, the average performance 
for the last five flights is visibly better than that of the first three flights. But the average is 
unduly influenced by the performance of participant #25. One of the participants in the NH 
group, #3 1 , offers a strange pattern of alternating increase and decrease of performance from 
T1 to T8. Few other participants in both groups show a visible decrease in their performance 
at different stages of their training sessions. Also, there seems to be no improvement in 
performance for both FIH and NH participants from T6 - T8. This could mean that perhaps 
our training task is too easy, and the number of flights is too many so that the participants 
have reached a plateau in their training and can’t be further trained in that scenario. 

The transition from training to evaluation going from T8 to El shows that most of the 
participants in both groups have performed worse in the first evaluation flight than in the last 
training flight. That is understandable due to a relatively more challenging task in the 
evaluation flights when compared with the training flights. For the trainees in the NH group 
the added challenge in the evaluation flight is the presence of mild turbulence. For the 
trainees in the HH group, the absence of hoops in the evaluation flight is another challenge 
together with the presence of turbulence. Surprisingly, in the transition from training to 
evaluation, the additional challenge for the HH group did not result in a larger decrease in 
performance than that in the NH group. This could be taken as a positive aspect of training 
with hoops as compared with training without hoops. 
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Figure 17 shows that the performance of the NH Group, on average, was worse than 

the HH group when transitioning from the last training flight to the first evaluation flight. 
The hypothesis was that a training intervention, such as tunnel in the sky, would be a better 
way to train pilots than would standard methods without a training intervention. Some 
evidence of this may be seen in the first evaluation flight (El) in which performance of the 
HH Group was slightly better than the NH Group (Figure 1 7). 
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Figure 13. Normalized performance of the participants in NH group. 
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Figure 14. Performances of the participants in the NH group. 
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Figure 15. Normalized performance of the participants in the HF1 group. 
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Training Flights Evaluation 
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Figure 16. Performance of the twelve participants in the hoop (HH) group. 

(Missing data: T5 for HH001, T8 for HH005, T5 HH024, (El, E2) from HH025, (T2, T3) for HH029, HH009 
data for T8, El, E2 is suspect; Not shown here T1 for HH001; T1-T3 for HH025) 


Average Performance Metric for Hoop and No Hoop flights 
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Figure 1 7. Average performance of the NF1 and HF1 groups 
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CHAPTER IV 

COMPARSION OF COMPUTED METRIC WITH CFI EVALUATION 

Introduction 

For the student pilot in a FAR Part 61 or FAR Part 141 flight school, ten flight hours 
of required training can be flown in a PCATD. However, the regulations state that a CFI 
must also be present to supervise the training (FAA 1997). Vruels and Obermayer . 
emphasize the need of automated performance measures on modern simulators as a substitute 
for direct evaluation of performance by an instructor. To address this goal for training of a 
landing maneuver, the present effort included a study of how closely a computer can emulate 
a CFFs evaluation of a student pilot. 


Method 

As mentioned above, the study of landing performance of novice pilot on MSFS 2002 
required all trainees to fly two evaluation flights under identical conditions irrespective of the 
strategy used in their training flights. These landing approaches were video recorded using 
the MSFS2002 capability so that the flights could be independently evaluated by three CFFs 
with a minimum of 5000 hours of experience (for details see Appendix E). 

The CFI’s were required only to grade the approach, and not the flare and touchdown. 
CFFs were not informed regarding the training intervention experienced by the pilots. They 
were asked to assign each flight a grade based on a 100-point scale. They were also 
requested to monitor three flight parameters namely, airspeed, rate of descent and runway 
alignment. Of those parameters they were to indicate the best and the worst controlled flight 
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parameters and the flight parameter most affecting the grading. The details of the instructions 
to the CFIs are given in Appendix D. 

Results and Discussion 

For the thirty -three evaluation flights graded individually by the three CFFs, 
correlations between the pairs of instructors’ grades were found to be rather low (r = 0.43, 
0.51, and .67; see Figure 18.) Therefore, the CFI scores were further analyzed for flights in 
which the CFIs agreed on both the best and the worst controlled parameters for a flight. 


Instructor Grade Correlation 



Figure 1 8. Correlation of instructor grades. 
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All three CFIs cited runway alignment as the best controlled parameter more often 
than the other two parameters (Figure 19). Also, rate of descent was identified as the worst 
controlled parameter more often by all three CFIs (Figure 20). All three instructors agreed 
on both the best and worst parameters for seven flights. For grading of the seven identified 
flights, despite their agreements, a high correlation was seen only between the grades of two 
CFIs (r 2 =0.89, Figure 21). The grades of the third CFI had low correlations with the grades 
awarded by either one of the other two CFIs (r 2 = 0.44 and 0.25, Figure 21). 


Best Controlled Parameter 



R of D RnwyAlign Airspeed 

Parameters 


Figure 19. Best controlled parameters as cited by the CFIs. 


Grade <w' Number of times cited 



Parameters 

ire 20. Worst controlled parameters as cited by the CFls. 



Figure 21 . Correlations between flight instructor scores of selected flights. 
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For the seven landing flights identified, the calculated composite error metric values 
were plotted on a graph separately against the grades awarded by every one of the three 
instructors as shown in Figure 22. For the composite error metric, strong and significant 
correlations were found with the grades awarded by every one of the three instructors (r 2 = 
0.82, t = 4.69, p < 0.01; r 2 = 0.77, t=4.15, p < 0.01; and r 2 = 0.63, t = 2.91, p < 0.05). 



Figure 22. Correlation between composite error metric and instructor scores. 


These correlations indicate the suitability of the composite error metric to be 
considered for use as a measure of performance on a straight-in landing approach. The fact 
that correlations between instructors on grading a landing approach were low, and that such 
grades correlated poorly with the performance metric based on RMSE in the flight 
parameters of rate of descent and runway alignment indicate the difficulties inherent in this 
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research area. The strategy of determining first the flights in which instructors agree on best 
and worst controlled parameters is a promising one. Such a strategy may be effective in 
narrowing the focus to fewer variables than what would normally be involved in grading a 
flight maneuver, and should lead to a better understanding of the variables involved in CFI 
evaluation of a maneuver. This strategy could then be applied to different maneuvers and 
perhaps weights could be applied differentially to different parameters depending upon the 
parameters most important in evaluation of that maneuver. 
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CHAPTER V 

CONCLUSIONS AND SUMMARY 

This thesis examined different training interventions for novice pilots in an attempt to 
develop techniques and performance measures that could be incorporated into an ‘intelligent’ 
training system. Effectiveness of training interventions were investigated by comparing 
performance on a level turn and straight-in approach maneuver flown by novices on a flight 
simulator with control groups flying the same maneuvers on the flight simulator. Metrics 
were defined to compute performance using flight parameter data of the maneuvers. 
Performance computed from flight parameter data of the treatment-group and control group 
flying the straight-in approach was also compared with certified flight instructor evaluation 
of the video data of the same flights to validate the metric used for computing the 
performance. 

The following conclusions are drawn from these studies: 

• Real Time Training (RT) with post flight feedback resulted in the best performance 
on the evaluation flight for a level turn as compared to no feedback or training with ARTT. 

• Above Real Time Training (ARTT) using a sequence of 1. 5/2.0 with post flight 
feedback was observed to be the next best strategy for a level turn. Thus, if time is of the 
essence then this training strategy is a promising approach. Further studies with more 
participants could validate this finding and determine whether 2.0 for all flights, or 1 .5 for all 
flights might be a more effective strategy. 
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• The particular post flight feedback provided to the participants flying the level turn, 
consisting primarily of graphical feedback which showed the actual ground track 
superimposed over the target ground track may be an effective form of feedback to 
implement in an "intelligent" flight simulator providing automated feedback. 

• For the landing task, no statistically significant difference was found between the 
group provided with visual out-of-the-window cues of hoops and the control group. This 
indicates that the additional visual cues did not add to the training value, perhaps because 
visual cues in the form of the runway and visual approach slope indicator lights were already 
available for this maneuver. 

• Poor correlations were observed between inter-CFI evaluations. Where the CFIs 
agreed on the best and worst controlled parameters, correlations were obtained between the 
computed performances using the performance metric developed in this study and the 
evaluations of the CFIs for the straight-in landing approach task. 

Future Studies 

• The two studies in this volume were meant to be steps on the way to creating an 
intelligent flight simulator for training novices. To that end, it would be interesting to look at 
the following: 

• To validate any of these training strategies, however, a) more participants need to be 
studied to determine if the effects observed are generalizable and robust, and b) 
studies assessing whether these methods produce better performance in the actual aircraft 


need to be undertaken. 
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• Conduct more studies to determine the reliability and robustness of the e ffect of 
ARTT as a strategy for training. Specifically, compare RTT with ARTT 1.5 on the turn 
maneuver, both using feedback. 

• Use additional feedback mechanisms to enhance the already provided graphical 
feedback, such as automated audio to correct for error. 

• A study of a combination of tunnel in the sky and ARTT as a training intervention 
might yield interesting information. 

• Each training intervention needs to be evaluated for more than one maneuver to 
determine its robustness. 


APPENDICES 
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APPENDIX A 


Visual Basic Scoring Module: 

Function to strip repeating data caused by pressing pause button. Also calculates 
the overall score for the familiarization flight. 

Function CalcScore() 

Dim TotalScore As Double 
Dim Count As Integer 
Dim Stuff 

' Module created by Bruce Heath 
' December 3 1 , 2003 

t 

' Module finds the beginning of relevent data and 
' then adds the scores for 1 80 rows (seconds) 

' The totaled scores are then averaged by 1 80 for 
' the average score 

t 

' Version 2 February 6, 2004 
' Complete February 12, 2004 

' Corrects the problem caused by pausing the simulator for more 
' than 300 seconds while the flight recorder is running 

row = 2 'First row of data 

col = 4 'Seconds column for Clock time 

'MsgBox Sheets("Data").Cells(3, 4).Value 

'This loop finds the starting point for the data 

Do Until (Sheets("Data").Cells(row, col).Value o Sheets("Data").Cells(row + 1, col).Value) 
row = row + 1 
Loop 

' Modification : If the row on the DATA sheet still has not reached the beginning of the 
' data after end of the fomulated cells on the rows on the SCORE sheet, 

' then find the beginning of the data on DATA and place the processed data on the SCORE sheet. 

' Continue to place processed data on the SCORE sheet until the DATA sheet meets an empty 
' row. 

I 

If ((Sheets("Score").Cells(row + 1, col). Value = "") And (Sheets("Data").Cells(row + 1, col). Value o "")) 
Then 

row2 = row 

Do While (Sheets("Data").Cells(row2 + 1, col). Value o "") 

'Fill in Data 

'Altitude 

Sheets("Score").Cells(row2, 1). Value = Sheets("Data").Cells(row2, 9).Value - 3000 
'Bank 

Jf (Sheets("Data").Cells(row2, 10).Value>= 180 And Sheets("Data").Cells(row2, 1 0). Value <= 360) 
Then • 

Sheets("Score").Cells(row2, 2). Value = 360 - Sheets("Data").Cells(row2, 10). Value 


63 


Else 

Sheets("Score").Cells(row2, 2).Value = Sheets("Data").Cells(row2, 10).Value 
End If 

'Airspeed 

Sheets("Score").Cells(row2, 3). Value = Sheets("Data").Cells(row2, 10). Value - 75 

'Altitude Raw Score 
With SheetsO'Score") 

If (.Cells(row2, 1). Value < 0) Then 
.Cells(row2, 4). Value = (.Cells(row2, 1). Value / 50) * -1 
Else 

.Cells(row2, 4). Value = .Cells(row2, 1). Value / 50 
End If 

'Bank Raw Score 

.Cells(row2, 5).Value = ,Cells(row2, 2).Value / 3 

'Airspeed Raw Score 
If (.Cells(row2, 3) < 0) Then 

.Cells(row2, 6). Value = (.CeIls(row2, 3) * -1 ) / 5 
Else 

.Cells(row2, 6).Value = .Cells(row2, 3) / 5 
End If 

Count = 1 

Do While (Count <= 3) 

'Assign Altitude Score 
'Assign Bank Angle Score 
'Assign Airspeed Score 

Select Case .Cells(row2, Count + 3). Value 
Case Is <= 1 

.Cells(row2, Count + 6). Value = 4 
Case Is <= 2 

.Cells(row2, Count + 6). Value = 3 
Case Is <= 3 

.Cells(row2, Count + 6). Value = 2 
Case Is <= 4 

.Cells(row2, Count + 6).Value = 1 
Case Else 

.Cells(row2, Count + 6). Value = 0 
End Select 
Count = Count + 1 
Loop 

'Three Score total for the row 

.Cells(row2, 10). Value = .Cells(row2, 7). Value + .Cells(row2, 8). Value + .Cells(row2, 9). Value 
'Average Score for the row 

.Cells(row2, 1 1). Value = .Cells(row2, 10).Value/3 
End With 
row2 = row2 + 1 


Loop 


End If 

' Now find the numeric score 

loopEnd = row +181 'Loop for the three minute time period 

scorecol = 1 1 'Column K average for each second on every row 
'Count off 1 80 seconds totaling the average scores 
displayrow = row 

Stuff = MsgBox(displayrow, , "Data Begins in Row") 

With Sheets("Score") 

Do Until row = loopEnd 

TotalScore = TotalScore + .CelIs(row, scorecol).Value 
row = row + 1 
Loop 

'The numeric score is this 
.Cells(7, 14).Value = TotalScore / 180 
'MsgBox TotalScore 
End With 
End Function 


APPENDIX B 

Informed Consent Form 


INFORMED CONSENT FORM 

DEVELOPMENT AND ASSESSMENT OF A NOVEL TRAINING PACKAGE USING SELF 
INSTRUCT! ON METHODS AND ABOVE REAT TIME TRAINING (ARTT) FOR BASIC 
MANEUVERING TASKS ON A FLIGHT SIMULATOR 


As an undergraduate student ofTuskegee University, you are invited to participate as a 
trainee in an experimental research study. The study wilt develop and evaluate a new 
program of training on a flight simulator. 

Your participation will require approximately two hours of flying on a simulator one day. 
You will take a seat as a pilot in a mockup cockpit located in the Flight Vehicle Design 
Laboratory on Tuskegee’s Campus. You will control the flight movements of the 
joystick, throttle and rudder pedals while watching the flight parameter changes on the 
simulated instrument panel Heads Up Display (HUD) and the moving icon of the airplane 
on the computet screens. Tlie available instructor will give you a set of instructions, allow 
you some time for familiarization and for asking questions, and provide feedback on your 
performance whenever necessary. 

No physical or medical testing risks are involved in the experiment. The experiment, 
however, is not an official pilot training. As a possible risk, some of the simulator flight 
techniques that you will learn may not he applicable in real flying. Moreover, as it 
happens in any learning exercise, you may need to overcome temporary disappointments 
on possible lack of progress in performance, especially when your flight would face a 
simulated crash. 

In general, the training will be an enjoyment and a learning experience. The data 
provided by you on a survey form and the data on your performance in the training will 
be coded to protect your confidentiality. Your participation is voluntary and you may 
withdraw from the experiment at any time. Your participation is voluntary and you may 
withdraw from the experiment at any time. Your participation according to your agreed 
upon schedule, however will be greatly appreciated by the investigators. 

'Hie successful completion of this research program will be a valuable contribution by 
Tuskegee University in improving the methods of training pilots on a simulator. 

For any questions or concerns, please feel free to contact the investigators on campus: Dr. 
Syed Ali at 727 - 8853, Dr. Marcia Rossi at 727 - 8830 or Dr. Muhammad Khan at 727 - 
8637, In the event of an unresolved grievance, you may also contact Dr. Stephen Sodekc, 
the Chairperson of the Human Subjects Review Committee (HRSC) at 727 - 8363. 

The available investigator will give you a copy of this form to keep. By signing below, 
you are agreeing to participate in this study. 


Signature ofParticipant/ Date 


Signature oflnvesiigator/ Date 



Participant Demographic and Data Form Turn Study 


Background Survey- Turn Study 


Code: 

Major: 

Sex: Male 

Female 


Age: 

Are you: Left Handed 

Right Handed 

Do you have any flight experience? 

Yes 

No 

If so how many hours (estimate)? 

How long ago was your most recent flight training, if any? 

How many hours a week do you engage in video/ computer games? 

light: 0-5 hours 

medium: 6-13 hours 

heavy: 13 or more hours 

What type of video j computer games do you play? 

sports 

fighting 

war 

_____ cards 

flight 

mystery 

other: If so what type? 

What type of controllers do you use? 

joystick 

control pad 

keyboard 

arcade 


Debriefing Form for Turn Study 


Subject Code: 

Ask them if they have any questions about what they were doing? 

1) Questions asked by participant and answers 


2) Would you describe what you did as {circle one) 

very fun 
somewhat fun 
neutral 

somewhat boring 
very boring 

3) Would you agree that the pre training instructions (before any flight) prepared 
you for the flight training sessions? 

strongly disagree 
disagree 
can't decide 
agree. 

strongly agree 

4) Would you describe the training sessions as (circle one) 

very difficult 
difficult 
neutral 
easy 

very easy 

5) Would you agree that the flight tasks (the first flying sets) prepared you for the 
flight test? 

strongly disagree 
disagree 

can't decide/ don't know 
agree 

strongly agree 


Tell them the purpose of the study, i.e., to determine if training at different ART 
levels for turning is more effective than RT training. Answer any questions. Ask them 
not to reveal the nature of the experiment to any others, as it may disrupt their 
performance. Include any comments they made that seem relevant, that might help 
in explaining their performance. 



APPENDIX C 

Participant Demographic, Data Form, Debriefing Form, Landing Study 

Background Survey- Landing Study 

Code: . 

Major: 

Sex: Maie 

___ Female 


Age: 

Are you: Left Handed 

Right Handed 

Do you have any flight experience? 

Yes 

No 

If so how may hours (estimated)? 

How long ago was your most recent flight training, if any? 

Do you have any flight simulator experience? Yes No 

Overall, how much exposure have you had to Microsoft Flight or Combat 
Simulator? 

How many hours a week do you engage in video/ computer games? 

Light: 0-5 hours 

Medium: 6-13 hours 

Heavy: 13 or more 

What type of video / computer games do you play? 

Sports 

Fighting 

War 

Cards 

Flights 

Other: 

What type of controllers do you use? 

Joystick 

Control Pad 

Key boa rd 

Arcade 

Have you ever belonged to an organization similar to the C.A.P. or Redtails? 

Yes If so what organization ? 

No 


Are you familiar with any of the following concepts, if so please check? 
Throttle Aileron Rudder 


Blank Page 


Debriefing For Landing Study 


Participant Code: 


1. Would you describe what you did as (circle one ) 

Very fun 
Somewhat fun 
Neutral 

Somewhat boring 
Very Boring 

2. Would you agree that the verbal instructions and the orientation 
instructions (before any flight) prepared you for the training flights? 
Strongly disagree 

Disagree 
Can't decide 
Agree 

Strongly Agree 

3. Would you describe the training sessions as (circle one) 

Very difficult 

Difficult 

Neutral 

Easy 

Very easy 

4. Would you agree that the training sessions prepared you for the 
flight tests (the last two flights without sound)? 

Strongly disagree 
Disagree 
Can’t decide 
Agree 

Strongly Agree 

5. Are there any questions about what you were doing? 


The purpose of the study: To determine if the use of cues (hoops and automated 
pilot) was effective for landing tasks. Please do not to reveai the nature of the 
experiment to any others, as it may disrupt their future performance. Thank you 
for your participation. 



APPENDIX D 

Instructions and Data form for Flight Instructors 
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APPENDIX E 
Instructor Experience 


Instructor 1 

Extensive experience in 
Flight simulation and flight 
testing procedures 


Instructor 2 

5500 plus flight hours 
including 4000 plus 
instructional hours 

ATP Certification 

Instructor 3 

7000 plus flight hours 
including 500 plus 
instructional hours 

ATP Certification 
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An Experimental Study of the Effect of Ou t-of-th e-Window Cues on Training 

Novice Pilots on a Flight Simulator 

M. Javed Khan, Marcia Rossi, Bruce Heath, Syed F. Ali, Marcus Ward 
Tuskegee University, Tuskegee AL 36088 


Abstract 

The effects of out-of-the-window cues on learning a straight-in landing approach and a 
level 360° turn by novice pilots on a flight simulator have been investigated. The 
treatments consisted of training with and without visual cues as well as density of visual 
cues. The performance of the participants was then evaluated through similar but more 
challenging tasks. It was observed that the participants in the landing study who trained 
with visual cues performed poorly than those who trained without the cues. However the 
performance of those who trained with a faded-cues sequence performed slightly better 
than those who trained without visual cues. In the level turn study it was observed that 
those who trained with the visual cues performed better than those who trained without 
visual cues. The study also showed that those participants who trained with a lower 
density of cues performed better than those who trained with a higher density of visual 
cues. 

Introduction 

With the increasing availability of low-cost powerful personal computers the 
development of ‘intelligent’ flight simulators for use by general aviation flight schools is 
becoming viable (Ludwig et. al. 2002, Remolina et. al. 2004)). Such simulators will be 
able to use the available computational power to incorporate not only traditional but also 
novel training techniques based on learning theories. Also, these simulators will have 
robust and realistic assessment of task performance to provide feedback thus making self- 
instruction effective as well as controlling the training to progress in the right direction. 
The motivation for developing ‘intelligent’ flight simulators of course is to reduce the 
cost of flight training. The FAA already has allowed a certain number of simulator hours 
on Personal-Computer based Aviation Training Devices (PCATD) to substitute for flight 
hours thereby reducing the cost of training. However, these simulator hours have to be 
supervised by a certified flight instructor (CFI). The main advantages of ‘intelligent’ 
flight simulators then would be to increase the number of simulator hours substituting for 
flight hours, effective utilization of simulator hours, increased transfer of training thereby 
further reducing flight hours to gain proficiency, and reducing/eliminating the need for a 
CFI during flight simulator training. 

To reach these and other such goals there are a number of areas which need to mature 
before the foil potential of such a flight simulator can be realized and used routinely. 
These include cognitive scaffolds such as aural, visual and environmental cues for 
learning various flight skills, performance evaluation metrics faithfully capturing certified 
flight instructors’ assessment techniques and seamless incorporation of simulator-based 
training in the flight training curricula. 
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The current research has focused on special out-of -the window (OTW) visual cues in 
a flight simulator to develop a pedagogical model for training novice pilots. The use of 
OTW visual cues in flight training is not a novel idea. Two such cues are (a) using a 
certain perspective of the runway to judge height/glide slope and (b) maintaining a far-off 
object on a certain location on the windscreen/canopy for learning to maintain glide slope 
or keeping a tree/fixed point on ground for 360° level turns. However, if cues are used 
without understanding their limitations, incorrect habits developed may result in 
potentially hazardous situations. Foyle et. al. [1992] have pointed to a number of such 
cues which have high saliency but low reliability as is pertinently pointed out in AIM 
Chapter 8-1-5 as well. 

A number of studies have evaluated various aspects of ‘perspective displays or the 
more commonly referred to as tunnel-in-the-sky concepts. These include the effects of 
egocentric and exocentric depictions, terrain texture, projected path cues, trajectory shape 
(Mulder 2003(1), Mulder 2003(2), Doherty & Wickens 2001, Sachs 2003). Such " 
perspective displays are now commercially available. However it must be noted that this 
concept is being implemented primarily in a heads-down display (HDD) mode. It has 
been reported that flying in the tunnel resulted in definite improvements in flight path 
control. Limited research has been conducted in evaluating its utility as a 
navigation/flight path aid in a heads-up display (HUD) mode. Fadden et. al.[2001 ] have 
reported their observations of the implementation of the ‘tunnel-in-the-sky’ or 
‘perspective’ displays as a HUD. They determined that it did result in improvements 
albeit with cognitive tunneling. This was exhibited in the form of inattention towards 
unexpected traffic outside the tunnel. They also conducted experiments to compare the 
HUD ‘tunnel’ with a HDD ‘tunnel’. They determined that a HUD implementation had 
slightly degraded perfonnance in vertical path control while resulting in better 
performance in lateral flight path control in comparison to a HDD implementation. 
However their overall assessment was that a HUD implementation would combine the 
advantage of a HUD and a perspective display. 

The advantages of the tunnel-in-the-sky motivated us to investigate a HUD 
implementation of such a visual cue as a cognitive scaffold to help learn flight path 
control by novice pilots. In fact such an implementation exists in the fonn of the ‘magic 
hoops’ of the landing lesson in the various versions of the Microsoft Flight Simulator 
(MSFS). 

Experimental Method 

The two maneuvers chosen to evaluate the concept were a straight-in landing 
approach and a 360° level turn. These two maneuvers were expected to capture the 
effects if any of path curvature for such a training concept. Each maneuver was flown by 
a different batch of novice pilots. Each batch consisted of a control group and a number 
of treatment groups. The treatments philosophy was based on the concept of gradually 
reducing the dependence on the scaffold. Two methods of fading were evaluated. The 
first method was to reduce the number of hoops (tunnel cross section outlines) as the 
training progressed. The other approach was to mix flights with hoops and without hoops 
in a certain sequence as the training progressed. Preliminary studies with the hoop- 
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reduction technique for a level turn exhibited the need for a certain minimum number of 
hoops on the flight path for it to be effective as a training aid. For the level-turn maneuver 
two hoop-densities were used to investigate the effectiveness of hoop density. The study 
then concentrated on evaluating the effectiveness of mixing flights with hoops and 
without hoops. 

The study was divided into two parts. The first part evaluated the effect of visual 
cues on a learning a straight-in landing approach, while the other part investigated a level 
turn. The participants were undergraduate students of an introductory psychology course 
who took part in the study for extra credit. They had no prior flight simulator or flight 
experience. The participants were screened for their aptitude to fly the simulator based on 
their performance in three straight and level flights. The selected ‘novice’ pilots were 
then randomly assigned to various treatment and control groups. 

Straight-in Landing Approach . 

The study consisted of three groups flying a straight- in landing approach in a 
Cessna 172. Each group flew 8 training flights followed by an evaluation flight more 
challenging than the training flight. The training flights were flown according to the 
sequence given in Table I. The participants then flew a straight-in approach with mild 
turbulence increasing their workload to maintain the glide slope. 

360 '- Level Turn . 

In this experiment the participants flew 8 training flights which consisted on a 
360° level turn with a 10° bank angle. The control group was not provided any cues 
while the two treatment groups flew some flights with visual cues and others without. 

The visual cue densities differed for the two treatment groups. The training flights 
sequence is given in Table II. After the training the participants then flew a more 
challenging flight by executing a 30° bank level turn. 

Performance Measures 

Various performance measures have been suggested to evaluate a pilot’s performance 
based on flight recorder data (Rantanen et al. 2001, Rantanen & Johnson 2005). These 
include standard deviation, root mean square error, number of deviations, total time 
outside tolerance and mean time to exceed tolerance. In this study the root mean square 
error (RMSE) was used as a performance measure. In case of the straight-in landing 
approach, the sum of the RMSE in air speed, rate of descent and runway alignment was 
used for evaluating perfonnance. For the level turn, RMSE from the required path as 
evidenced by bank angle and altitude loss determined the perfonnance of the participants. 
The perfonnance of all participants in each group was averaged to compare between 
group perfonnances. 

Experimental Setup 

The experimental setup was a Microsoft Flight Simulator (MSFS) 2002 
environment running on four PCs networked using the IPX/SPX gaming protocol. Three 
out-of-the- window views were driven by the three PCs while the fourth computer was 
used to display the synthetic instrument panel. Cougar a joystick and throttle and CH4 
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rudder pedals were used for controls. Flight parameters of interest were written to file 
every second for later analysis. The landing approach study used the canned landing 
lesson in MSFS2002 suitably modified to remove instructor audio and control inputs. The 
level turn study used gmax to render the visual cue hoops along the turn path. These 
hoops were included as add-on scenery in MSFS 2002. The flights were designed to start 
at the same location in a trimmed condition. 

Results & Discussion 

The study revealed the following aspects of the influence of out-the-window 
visual cues. 

Straight-in Landing Approach Task 

A summary of the task performance is shown in Table III. As can be noted, 
providing visual cues throughout the training regimen (H-group) did not help the 
participant in learning to fly the simulator. This is primarily because the participants 
probably were more concerned about flying through the hoops and hence looking OTW 
rather than also scanning the instrument panel to ensure that the flight parameters for the 
task were being maintained. In case of the participants who trained without the visual 
cues (NH-group) performed much better as they had probably learned to divide attention 
between the OTW view so as to maintain runway alignment as well as scan the 
instrument panel to maintain the flight parameters for the task. The participants who 
trained using a mix of the flights with and without visual cues (FH-group) though in the 
beginning exhibited similar performance as the H-group were most probably able to leam 
to divide attention effectively and maintain the flight parameters and took the hoops as a 
challenge to fly through. 

Level 360 s . Turn 

The performance in the level turning task is summarized in Table IV. It can be 
observed that visual cues had a major influence on the perfonnance in the evaluation 
flight. The performance improved for both the cue-density methods as compared to the 
group who trained without visual cues. This observation is consistent with the 
observation of the landing study. The perfonnance was observed to be better for the 
lower density method as compared to the higher density method. This is attributed to the 
increased time between hoops thereby forcing the participant to look inside the cockpit at 
the flight parameters required for the task. With the increased hoops density the tendency 
to ‘chase’ the hoops and not paying enough attention to the flight parameters most 
probably was the cause for the relatively lower performance. 

Table I: Treatment Sequences: Straight-in Landing Approach 

H: Flight with Hoops; NH: Flight with No Hoops; Evaluation Flight: E 


Group# 1 (Control) 
(NH) 

Group#2 (H) 

Group#3 (FH) 

NH 

H 

H 

NH 

H 

H 
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NH 

H 

H 

NH 

' H 

NH 

NH 

H 

NH 

NH 

H 

H 

NH 

H 

H 

NH 

H 

NH 

E 

E 

E 


Table II: Treatment Sequences for a 360 A Level Turn 

H: Flight with Hoops; NH: Flight with No Hoops; Evaluation Flight: E 


Group# 1 (Control) 

Group#2 (36 Hoops) 

Group#3 (18 Hoops) 

NH 

H 

H 

NH 

N 

N 

NH 

H 

H 

NH 

N 

N 

NH 

N 

N 

NH 

H 

H 

NH 

N 

N 

NH 

N 

N 

NH 

N 

N 

E 

E 

E 


Table III: Composite Error for Straisht-in Landins Approach 


Group 

Error 

Control (No hoops) 

25.5 

All training flights with hoops (H-group) 

32.1 

Faded-cues training flights (FH-group) 

22.2 


Table IV: Error for Level 360- Turn 


Group 

Error in Bank Angle 

Error in Altitude 

Control (No Hoops) 

17.99 

233.1 

Low Density Hoops 

10.12 

109.5 

High Density Hoops 

11.22 

170.3 
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Correlating Computed and Flight Instructor Assessments of 
Straight-In Landing Approaches by Novice Pilots on a Flight Simulator 

Bruce E. Heath (bheath35@hotmail.com), M. Javed Khan (mjkhan@tuskegee.edu), 
Marcia Rossi (rossi@tuskegee.edu), Syed Firasat Ali (sfali_tusk@yahoo.com) 

Tuskegee University, Tuskegee, AL 36088 

The rising cost of flight training and the low cost of powerful computers have resulted in 
increasing use of PC-based flight simulators. This has prompted FAA standards 
regulating such use and allowing aspects of training on simulators meeting these 
standards to be substituted for flight time. However, the FAA regulations require an 
authorized flight instructor as part of the training environment. Thus, while costs 
associated with flight time have been reduced, the cost associated with the need for a 
flight instructor still remains. The obvious area of research, therefore, has been to develop 
intelligent simulators. However, the two main challenges of such attempts have been 
training strategies and assessment. The research reported in this paper was conducted to 
evaluate various performance metrics of a straight-in landing approach by 33 novice 
pilots flying a light single engine aircraft simulation. These metrics were compared to 
assessments of these flights by two flight instructors to establish a correlation between 
the two techniques in an attempt to detennine a composite perfonnance metric for this 
flight maneuver. 

This research was supported by NASA Ames Award CTS-0320305 and NASA Dryden 
Grant NAG4-226. 


INTRODUCTION 

With the advent of faster computers, it has 
become possible and cost effective to use 
personal computer-based aviation training 
devices (PCATD) to provide supplemental 
training for pilots. Such devices are 
regulated by the Federal Aviation 
Administration (FAA). The airplane or flight 
training device (FTD) flight-hour training 
time required for an instrument rating may 
be reduced by using PCATDs that have been 
detennined to meet acceptable FAA 
standards. For the student pilot in a Part 61 
or Part 141 flight school, 10 of the total 
allowable flight-hours in a FTD can be 
flown in a PCATD. However, the 
regulations state that a certified flight 
instructor (CFI) must also be present (FAA 
1997). Vruels and Obermayer (1985) 


emphasize the need of automated 
perfonnance measures on modem simulators 
as a substitute for direct evaluation of 
performance by an instructor. One strategy 
to address this goal is to detennine how 
closely a computer can emulate a CFI’s 
evaluation of a student pilot. Therefore, an 
investigation of different computed 
evaluation techniques is needed to compare 
against instructor evaluations. 

Rantanen and Talleur (2001) have suggested 
the following five primary measures for 
pilot perfonnance from the flight data 
recorder for nine flight parameters: (1) 
Standard deviation. A small standard 
deviation is indicative of good performance. 
(2) Root mean square error (RMSE) which 
summarizes the overall error. (3) Number of 
deviations, which is a measure of 
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occurrences of the aircraft staying outside 
the prescribed tolerances. (4) Time outside 
tolerance is the cumulative time the aircraft 
spends outside prescribed tolerances. (5) 
Mean time to exceed tolerance is computed 
from the rate of change between successive 
data points and the aircraft’s position 
relative to a given tolerance. Heath and Crier 
(2003) used computerized scoring and 
certified flight instructors (CFI) to attempt to 
devise a computed performance measure for 
a level turn. A performance index based on 
deviations from the required heading, 
altitude and airspeed was compared with the 
instructors’ letter grades. One of the 
challenges of such comparisons is inter-rater 
reliability of the flight instructors. 

The objective of the current study was to 
explore the possibility of determining 
computed measures for a landing task by 
novice pilots on a flight simulator which 
would realistically depict their performance. 
The approach there has been that if these 
performance measures correlate with 
assessments of the task by CFIs that these 
measures then can be used routinely to 
inform student pilots of their performance. 
This study chose to use the standard 
deviations of the errors in flight parameters 
to represent pilot performance. A straight-in 
landing approach was chosen as the 
maneuver to be studied. 

METHOD 

Equipment 

The experiment was performed at the 
Aerospace Science Engineering Department, 
Tuskegee University. The flight simulation 
environment consisted of four computers 
that were all running Microsoft Flight 
Simulator 2002 (MSFS 2002). The 
computers were connected by LAN using a 
Netgear Fast Ethernet Switch. Three of the 


computers were used for out of the window 
(OTW) views. One computer was used as 
the main computer. Communication 
between the computers was made using 
FSUIPC; a third party software that runs 
over IPX/SPX. The OTW view was created 
using Wideview software; also third party. 
Data was collected into Excel spreadsheets 
using a third party flight data recorder. All 
of the computers run Windows XP 
Professional. The OTW view computers are 
all Gateway E series computers with the 
following specifications: Pentium 4 
2.60GHz processor Hyper-Threading 
Technology with 512KB cache, 512 MB 
ram, bus speed 800 MHz and memory speed 
of 333 MHz. An NVIDIA GeForce FX5200 
is the video card used for the OTW view 
computers. The master computer has the 
following specifications: Pentium 4 2.4GHz 
processor with 512 KB cache, 512 MB ram, 
bus speed 133 MHz. The video card for the 
master computer is a NVIDIA GeForce Ti 
4600 with 128MB memory. A Thrustmaster 
HOTAS joystick, a Cougar throttle and CH 
Products rudder pedals were used. 

Procedure 

Three CFIs were asked to individually view 
and evaluate 33 video recordings of straight- 
in landing approaches made by novice pilots 
on the flight simulator. The pilots were 
students at Tuskegee University who had 
had little or no experience flying an airplane 
or simulator. Two of the flight instructors 
also had airline transport pilot (ATP) 
certification with one CFI having 7 1 00+ 
flying hours with 580+ instructional hours 
while the other CFI had 5800+ flying hours 
with 4000+ instructional hours. The third 
CFI had extensive experience in certification 
requirements and flight testing and flight 
simulation. 
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The task given to the pilots was to maintain 
75 knots airspeed, descend at 500 feet per 

minute, and head towards and line up with 
the centerline in a straight-in approach to the 
Bremerton International Airport in the 
Seattle, WA area. The aircraft being flown 
was a Cessna 1 72. The flight scenario was a 
modified version of a landing lesson from 
the “flight school” portion of Microsoft 
Flight Simulator, MSFS 2002. The 
approaches made by the pilots were 
recorded and sent to CFIs for evaluation. 

The instructors were asked only to grade the 
approach, and not the flare and touchdown. 
Further, they were told to turn the sound off 
of the recording, and to disregard any text 
messages that may have appeared on the 
screen. They were asked to evaluate the 
flights after the ‘autopilot’ light was turned 
off. The CFIs were asked to assign each 
flight a grade based on a 100 point scale. 
They were also asked to monitor three flight 
parameters namely, airspeed, rate of descent 
and runway alignment and record the best 
and worst controlled flight parameters and 
the flight parameter most affecting their 
grading. 

The various flight parameters including 
airspeed, heading and altitude were written 
to an Excel file with a 1HZ frequency for 
each flight. Also certain other events such as 
‘autopilot status’, ‘aircraft on ground’ were 
also written to the file. The data for each 
flight was then processed to determine the 
error in airspeed, rate of descent and runway 
alignment. The standard deviations for these 
errors were then calculated. 


RESULTS & DISCUSSION 

The perfonnance measures investigated 
were the standard deviation of the errors in 
runway alignment, rate of descent, airspeed 
and linear combinations of these deviations. 


Pearson Product Moment Correlations were 
calculated between these measures and the 
individual instructor scores. Weak 
correlations (r 2 = 0.263, 0.09, 0.12 for rate 
of descent error, r 2 = 0.08, 0.1, 0.05 for 
runway alignment error, r = 0.14, 0.19, 0.1 
for airspeed error) prompted a detailed 
analysis of the instructors’ grading. 

Correlations between the pairs of instructors 
grades were calculated and found to be only 
moderate (r 2 = 0.4334, 0.5054, 0.4334). 
Therefore, the instructor responses were 
analyzed to determine whether there was 
consistency in scoring procedures not 
captured in the grades. The frequencies of 
citing the best and worst controlled 
parameters are shown in Figs. 1(a) and 1(b). 
As can be seen, all three instructors most 
frequently cited runway alignment as the 
best controlled parameter while citing rate of 
descent as the worst controlled parameter. In 
addition, one instructor also cited airspeed 
almost as frequently as the other two 
parameters as the worst controlled 
parameter. This suggested that a composite 
of the standard deviations of runway 
alignment error, rate of descent error and 
airspeed error to be a candidate error metric. 

Seven flights were identified for which all 
three instructors agreed on both best and 
worst controlled parameters. Correlations 
between the instructor grades and the 
composite error metric (a linear combination 
of the standard deviations of errors in 
runway alignment, rate of descent and 
airspeed) were calculated for these flights 
(Fig. 2). A strong and significant correlation 
was found for all three instructors (r 2 = 
0.8153, t = 4.6878, p < 0.01, r 2 = 0.7749, t = 
4.14877, p < 0.01, r 2 = 0.628, t = 2.91, p < 
0.05). Correlations between pairs of 
instructor grades were also calculated for 
these seven flights (Fig. 3). It can be seen 
that despite their agreement on the best and 
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worst controlled parameters, one of the 
flight instructor’s grades was weak to 
moderately correlated with the other two (r 2 
= 0.8896, 0.44, 0.2517). 

In conclusion the following observations are 
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(a) Inconsistency in grading between 
instructors is a probable cause for 
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grading and the error metric. 

(b) Despite agreement on best and worst 
controlled parameters, the weak to 
moderate correlation of one 
instructor’s grades with the others 
suggests that the instructors’ grades 
may be influenced by other factors. 

FUTURE RESEARCH 

• Study the effect of student pilot 
experience on inter-CFI correlation. 

• Evaluate the influence of student 
pilot experience on correlation of 
computed performance measures 
with CFI-grades. 

• Investigate other performance 
measures to determine most robust 
measure. 
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Abstract 

A multi-disciplinary and inter-disciplinary education is increasingly being emphasized for 
engineering undergraduates. However, often the focus is on interaction between engineering 
disciplines. This paper discusses the experience at Tuskegee University in providing inter- 
disciplinary research experiences for undergraduate students in both Aerospace Engineering 
and Psychology through the utilization of a low cost flight simulation environment. The 
environment, which is pc-based, runs a low-cost of-the-shelf software and is configured for 
multiple out-of-the-window views and a synthetic heads down display with joystick, rudder 
and throttle controls. While the environment is being utilized to investigate and evaluate 
various strategies for training novice pilots, students were involved to provide them with 
experience in conducting such interdisciplinary research. On the global inter-disciplinary 
level these experiences included developing experimental designs and research protocols, 
consideration of human participant ethical issues, and planning and executing the research 
studies. During the planning phase students were apprised of the limitations of the software 
in its basic form and the enhancements desired to investigate human factors issues. A number 
of enhancements to the flight environment were then undertaken, from creating Excel macros 
for determining the performance of the ‘pilots’, to interacting with the software to provide 
various audio/video cues based on the experimental protocol. These enhancements involved 
understanding the flight model and performance, stability & control issues. Throughout this 
process, discussions of data analysis included a focus from a human factors perspective as 
well as an engineering point of view. 

Introduction 

The ‘Programs Outcomes and Assessment’ detailed in Criterion 3 of the ABET2000 
document [1] for accreditation of engineering programs is a template for educating well- 
rounded engineers. This criterion is a consequence of an awareness of the fact that today’s 
graduates need to have skills for success and competitiveness in a global and highly 
interactive work environment. In order to fulfill the requirements of this criterion, educators 
are now consciously designing and delivering their educational materials [2]. 

Interdisciplinary research is considered to be an effective method for exposing 
students to the skills outlined in Criterion 3(a) - (k) of Reference [1], Such a research 
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environment when properly structured provides the students with opportunities to develop 
collaborative, technical and communication skills. The added advantage is that students 
actually are able to see a practical application of the skills they are acquiring and have a 
sense of accomplishment. The ‘Calpoly Interdisciplinary Monarch Butterfly Nuptial Flight 
Research’ [3] is an interesting example of interdisciplinary research in which students of 
behavioral ecology, molecular and cellular biology statistics and aeronautical engineering are 
attempting to determine characteristics for a successful mating. The Mind Project [4] of 
Illinois State University is designing ‘simulated persons’ using artificial intelligence, robotic 
devices and is although aimed at cognitive research has involved students from a variety of 
departments. 

The authors have been conducting research at Tuskegee University in the area of 
flight simulator based training methodologies. These investigations have been directed 
towards novice pilots as the volunteers for the research are students with no flying experience. 
Some of the factors that have been studied are the impact of above-real time (a scenario in 
what events are presented at a higher speed than normal) training strategies, self-instruction 
and team interactions etc. [e.g. 5,6], Currently an interdisciplinary research effort towards 
determining training strategies utilizing a low-cost flight simulation environment for ab-initio 
or novice pilot training and minimizing flight instructor interventions is being undertaken. 

This research has provided excellent interdisciplinary exposure opportunities for 
students from aerospace engineering and psychology majors [7, 8]. Interdisciplinary research, 
however have many unknowns and need to be carefully structured for students involvement. 
Some of these pertain to working attitudes, project concepts and plans, specialist jargon, 
execution methodologies etc. For example Reference 9 notes that ‘established means of 
facilitating working relationships among social and computer scientists are currently lacking’. 

This paper details the approach taken to successfully involve students. 

The Team & Tasks 

The objective of the investigation was to detennine an effective training strategy on a 
flight simulator for learning to land a light aircraft using a straight-in approach. The faculty 
research team consisting of the authors identified a group of psychology and aerospace 
engineering majors students interested in the research who were invited to the preliminary 
planning meetings. The research meetings served an important purpose. Students were 
exposed to interdisciplinary terminologies, concepts and interactions between the disciplines. 
For example the engineering students were exposed to the subtleties of human responses, 
while the psychology students gained a better understanding of time needed to setup physical 
hardware and develop software patches etc. They also realized the constraints due to 
limitation of resources, and time. 

These initial meetings established the tasks and timelines to achieve the research 
objective. The students were then given various responsibilities based on the following tasks: 

(a) Establishing the flight simulation environment 

(b) Determining orientation, training and evaluation maneuvers 
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(c) Coding automatic performance grading 

(d) Designing the experiment 

(e) Recruiting volunteers 

(f) Establishing experiment protocols 

(g) Conducting the experiment 

(h) Analyzing & evaluating the data 

The first objective for the aerospace engineering students was to establish the low 
cost flight simulation environment. During the planning meetings the various required 
characteristics of the environment were discussed. The students brought some useful 
observations to the table regarding maintaining interest of volunteers in the research. These 
included the importance of a realistic and interesting flight simulation environment. This was 
based on their earlier experiences with a simulator which had a bland desert scenery. 

The establishment of the flight simulation environment then was an exercise in 
systems engineering as the objective was to use off-the-shelf software and hardware. The 
public domain FlightGear Flight Simulator and the commercial Microsoft FS2002 
Professional software were evaluated by the students to determine its ease of use, sensitivity, 
realism etc. For the current investigation the MSFS2002 was detennined to be a suitable 
engine. However, to use it effectively a number of enhancements were needed. One of the 
requirements for the enviromnent was to have a multiple monitor out of the window display. 
An extensive internet search was conducted by the students to determine availability of 
software extending the capability of the MSFS2002. The WideView [9] software was found 
to provide exactly this capability. The multi-monitor capabilities of WindowsXP software 
were used for configuring the various parts of the synthetic instrument display. Since one of 
the objectives of the research was to minimize the involvement of a ‘flight instructor’ in the 
learning process, an automated performance grading had to be established. This required the 
comparison of the actual with the desired flight parameters; hence a data extraction software 
was needed to interact with the simulation engine. Again an internet search was conducted 
and the ‘FltRec82’ [10] was determined to be an appropriate software. However the data 
extracted by this utility had to be manipulated to determine the performance of the ‘pilot’. An 
excel macro was therefore written by the students to calculate and provide a quick answer to 
the student researcher as to whether the ‘pilot’ had passed or failed the flight maneuver. The 
students were exposed to the various performance metrics used in previous investigations in 
which the students were involved. During these previous investigations the students had 
conducted a comparative study of various grading strategies with flight instructors grading [8] 
exposing them to statistical analyses. While the aerospace engineering students were at ease 
with the discussion of the rationale for including certain flight parameters while excluding 
others, the psychology students were exposed to practical applications of concepts like that of 
kinetic and potential energy interchanges which they had studied in physics courses. 

The research team meanwhile considered various candidate training methodologies 
for investigation. The impact of the use of visual cue of flying through a ‘tunnel in the sky’ 
on the landing approach versus instrument scanning technique was chosen for investigation. 
The impact of ‘fading’ of the visual cues was also included. The reasoning behind the 
planned sequencing of flights, and fading was explained to the team along with the design of 
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the experiment. The aerospace engineering students were thus exposed to the theory of 
design of experiments and became familiar with concepts like ‘between-subjects’ and ‘within 
subject’ designs. 

Since the research was to be conducted with volunteers who had no flying or flight 
simulator experience, effective protocols for conduct of the experiment had to be designed. 
These included standardized briefings on simulator operation, using the post- flight feedback 
on performance, orientation flights, training and evaluation maneuvers. This process required 
intensive teamwork between the engineering and psychology students and involved role- 
playing (of the volunteer and pre-experiment briefer etc.) to ensure standardized, repeatable 
and smooth conduct of the experiments. During this process the student researchers were 
exposed to the important aspect of sensitivity to the volunteers’ response to the evaluation of 
their performance/skill. The students then presented their protocols and demonstrated the 
experimental procedure to the faculty team members. 

The requirement of standardized orientation, training and evaluation flights scenarios 
was met by understanding the Adventure Basic Language on the MSFS2002 so as to 
appropriately modify the landing lessons already programmed in the software. Also, the 
orientation flights which were primarily straight & level flights to screen the volunteers 
required an understanding of the interactions between altitude, throttle setting and speed to 
establish the desired flight parameters. This provided an opportunity for the psychology 
students to understand the basic aspects of flight mechanics. 

As the investigation would utilize volunteers, the University’s requirements for 
research involving human subjects had to be fulfilled. The students were thus exposed to the 
ethical aspects of research. Details of the research had to provided to the University for 
approval and the students realized that a well thought out research plan was essential as after 
the approval of the plan by the University, any changes would again have to be referred to 
the Human Subjects Committee causing delays in the investigation. 

Fonns were designed by the students to collect demographics and data organizing 
coding was established. The students then conducted final ‘dry runs’ to ensure the 
experimental procedures correctness and determined the durations for the various phases of 
the experiments. Flyers for soliciting volunteers for the research were designed and 
distributed. An elaborate schedule was then worked out by the student team allocating time 
slots to the volunteers and the student researchers. Data was then collected on a number of 
volunteers. 

Summary & Conclusions 

An interdisciplinary team of aerospace engineering and psychology majors was 
involved in an interdisciplinary research project. The research provided the students an 
opportunity to function as a closely knit team. They were exposed to: 

(a) Participation and interaction as a research group 

(b) Establishing tasks for achieving research objectives 

(c) Time and effort management 


Proceedings of the 2004 American Society’ for Engineering Education Annual Conference & Exposition 5 
Copyright © 2004, American Society for Engineering Education 



(d) Interdisciplinary academic areas 

(e) Literature searches 

(f) Formal presentations of results 

The team successfully met the objectives set forth by the faculty mentors. The main 
elements of success were primarily a high level of motivation of the students due to the 
opportunity of learning of concepts not covered in their major areas, being given 
responsibility of tasks with practical impact on the investigation and planning and actual 
conduct of the experiments under minimal supervision. However, it is essential that for 
interdisciplinary student research teams, the faculty mentors have a well-structured research 
plan so as to minimize overwhelming of the students. Regular meetings to provide well 
defined tasks are an important element of the structure. 
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ABSTRACT 

The use of Post-Flight Feedback (PFFB) and 
Above Real-Time Training (ARTT) while 
training novice pilots to perform a 
coordinated level turn on a PC-based flight 
simulator was investigated. One group 
trained at 1.5 ARTT followed by an equal 
number of flights at 2.0 ARTT; the second 
group experienced Real Time Training 
(RTT). The total number of flights for both 
groups was equal. Each group was further 
subdivided into two groups one of which 
was provided PFFB while the other was not. 
Then, all participants experienced two 
challenging evaluation missions in real time. 
Performance was assessed by comparing 
root-mean-square error in bank-angle and 
altitude. Participants in the 1. 5/2.0 ARTT 
No-PFFB sequence did not show 
improvement in performance across training 
sessions. An ANOVA on performance in 
evaluation flights found that the PFFB 
groups performed significantly better than 
those with No-PFFB. Also, the RTT groups 
performed significantly better than the 
ARTT groups. Data from two additional 
groups trained under a 2.0/1 .5 ARTT PFFB 
and No-PFFB regimes were collected and 


combined with data from the previously 
studied groups and reanalyzed to study the 
influence of sequence. An ANOVA on test 
trials found no significant effects between 
groups. Under training situations involving 
ARTT we recommend that appropriate 
PFFB be provided. 

INTRODUCTION 

Advancements in computer technology have 
made the use of computer-based simulators 
and trainers more feasible for research 
investigating factors related to pilot training 
effectiveness. One of the training strategies 
utilized for skills acquisition is that of 
ARTT in which training skills are acquired 
on a real-time simulator when it is modified 
to present events faster than normal. The 
motivating factors for ARTT can be traced 
back to Kolf (1973) who noted that, 
"regardless of type or amount of pre-flight 
simulator training accomplished by the pilot, 
the actual flight appears to take place at a 
much faster time frame than real time" and 
Hoey (1976) who reported that the mental 
state of test pilots operating remotely piloted 
vehicles can be approximately simulated 
without stressful conditions by increasing 
the simulated rate of time passage. More 
recently, Crane and Guckenberger (1997) 
reported that pilots trained using ARTT 
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performed emergency procedures and 

defeated bandit aircraft significantly faster 
than pilots trained in real time. Rossi et al. 
(1999) trained university students on a 
gunneiy task to compare RTT and ARTT at 
1.5 times real time. The students trained in 
ARTT performed on test trials as well as 
students trained in RTT, although the ones 
trained in ARTT spent less clock time. 
During training, the performance of students 
in ARTT was depressed compared to those 
trained in RT. The authors suggested that 
using ARTT as top-off training after RTT 
might result in more effective training. 
Williams (1999) also observed that ARTT as 
top-off training after RTT offers better 
training in comparison to ARTT alone or 
RTT alone in a similar gunnery task. Ali, 
Guckenberger, Rossi, and Williams (2000) 
addressed the use of ARTT for training of 
pilots to perform basic flight maneuvers. 
They classified the flying maneuvers with 
reference to Fitts and Posner's (1967) model 
that recognizes different stages of skill 
acquisition as cognitive, associative, and 
autonomous. Ali et al. (2000) observed that 
using ARTT at the cognitive stage was 
beneficial for acquiring skills in straight- 
and-level flight. However, as climb and 
descent and level turns are relatively more 
complex than straight-and-level flying, they 
propose that ARTT is beneficial at the 
autonomous stage or as top-off training after 
RTT. 

Self-instruction through the use of feedback 
is increasingly being incorporated in 
computer-based learning. Proctor and Dutta 
(1995) provide a comprehensive discussion 
of the influence of feedback on motor skill 
acquisition. Ali et al. (2000) used two forms 
of feedback; first, automated in-flight aural 
cues that signaled to the pilot trending out of 
prescribed tolerances in altitude and heading 
and second, post-flight feedback in the form 
of strip-charts consisting of a graphical 
comparison of altitude, heading, airspeed, 
and other parameters during flight with the 
target values/tolerances. They observed that 
these two feedback strategies in general 
improved the performance. 


In summary, previous studies have 

determined that the efficacy of training 
depended on types of feedback, the type of 
piloting tasks, and the use of ARTT. The 
interaction of these parameters has not been 
systematically studied for training of novice 
pilots. 

Objectives 

The current investigation consisted of two 
related studies that attempted to: 

(a) understand the influence of different 
sequences of ARTT values, and 

(b) determine the adequacy of PFFB. 

EXPERIMENTAL SETUP 

Mock setup of a partial cockpit housed in 
the Flight Vehicle Lab at Tuskegee 
University was used as the pilot training 
station. This setup provides an out-the- 
window (OTW) panoramic view on three 
monitors with a heads-up display (HUD) on 
the center monitor. The heads-down display 
(HDD) of a conventional instrument panel 
was on a fourth monitor located below the 
OTW monitors. The four monitors are 
controlled by a Quantum 3D Heavy Meta! 
Computer having two Pentium II 400 MHz 
processors, 400 MB RAM, three extra 
display cards for Open GVS, based graphics, 
and a Sound Blaster audio card. The 
computer was configured by SDS 
International, Orlando, Florida and runs the 
Lite Flite version 3.3 (199) flight simulation 
software by SDS International. Lite Flite 
offers flight simulation of several aircraft 
including a Predator unmanned air vehicle 
(UAV) which was used in these studies. The 
controls include a Saitek X36F joystick, a 
Saitek X35T throttle, and CH rudder pedals. 

EXPERIMENT 1 

The purpose of this experiment was to 
investigate the effects of increasing values 
of ARTT in comparison to RTT, and the 
effects of PFFB vs. no PFFB during the 
training of a standard two-minute turn with 
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novice pilots. As previous studies have 
suggested ARTT to be useful for top-off 
training, the use of increasing values of 
ARTT may also prove beneficial. 

Participants 

Twenty undergraduate college students 
enrolled at Tuskegee University who had 
little or no prior flying experience served as 
novice pilots in exchange for course credit. 

Experimental Design 

This experiment consisted of a 2 X 2 X 2 
mixed factorial design. The between 
subjects variables were the type of training 
(RTT vs. ARTT) and feedback condition 
(PFFB vs. No-PFFB). The within subjects 
variable was the two evaluation flights. 

There were five participants in each group, 
with the exception of four in a RTT, no 
feedback (No-PFFB) group. 

Procedure 

All participants experienced the following 
segments: Orientation to Simulator Controls 
and Functions, Demonstration Flight, 
Familiarization Flights, Training Flights, 
and Evaluation Flights. 

Orientation to Simulator Controls and 
Functions. Demonstration and 
Familiarization. During the orientation 
process, participants were informed of the 
basic control surfaces of the aircraft, their 
functions, and the movements associated 
with each control surface. The participants 
then were given an overview of the locations 
and functions of the joystick, rudder pedals, 
and the throttle located in the mock cockpit. 
Then, participants were instructed in the 
location and functions of the following 
displays on the HUD: altimeter, radio 
altimeter, airspeed indicator, heading tape, 
artificial horizon, pitch ladder, and clock. 
They were also instructed on the location 
and functions of the following instruments 
on the HDD: artificial horizon and the 
vertical velocity indicator. Because the turn- 
and-slip indicator on the HDD was not 


functioning properly, the use of the HUD to 
determine if the aircraft was making a 
coordinated turn was explained. 

Participants observed the experimenter fly 
one 3-minute flight, while explaining the 
controls and operation of the aircraft. 
Participants flew four flights of 3-minute 
duration each in which they were told to 
maintain airspeed of 129 knots, a heading of 
360 degrees, and an altitude of 5,000 feet. 
Participants who achieved a score of 2.0 
(grade based on comparing parameter values 
in the flight with the parameter values and 
tolerances in an input file) on at least one 
familiarization flight continued into training. 
Twenty participants, who achieved the 
passing grade continued on to the training 
phase. 

Training and Evaluation . The experiment 
consisted of a training phase and an 
evaluation phase. The training mission 
consisted of a coordinated 1 80° turn with a 
1 0° bank angle while the evaluation task was 
an S-turn with a bank angle of 30°. 

However, a loss of situational awareness 
was observed in most of the participants 
after the first leg of the S-turn. Thus, only 
the first leg of the S-turn was evaluated for 
analysis purposes. Each group conducted 
their training in the following sequence 
immediately followed by two evaluation 
flights in real time. 

Group I: Five missions in 1 .5 ARTT then 
five missions in 2.0 ARTT with No-PFFB 

Group II: Five missions in 1 .5 ARTT then 
five missions in 2.0 ARTT with PFFB 

Group III: Five missions in RTT then five 
missions in RTT with No-PFFB 

Group IV: Five missions in RTT then five 
missions in RTT with PFFB 

Feedback. Participants in the No-PFFB 
groups received no feedback during or after 
flights. If they asked questions about the 
task, they were reread the relevant 
instructions. Participants in the PFFB groups 
received verbal feedback after each flight 
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consisting of information about errors in 

manipulating the controls and in focusing on 
the wrong instruments. After the second, 
fourth, sixth, eighth, and tenth flights, 
participants in the PFFB groups were also 
shown a printout of the desired ground track 
and the pilot's actual ground track for 
comparison. Then, the instructor explained 
possible reasons for the deviations in 
performance 

Performance Measure . Performance of the 
pilots was assessed by comparing a ‘root- 
mean-square’ error in the bank-angle, and 
altitude (normalized by the radius of turn) 
calculated every three seconds of flight and 
averaged over the duration of the flight. In 
general the error in velocity should also 
have been included but it was observed that 
once the pilot initiated a turn, there was no 
attempt to manipulate the throttle, as a 
consequence of which the changes in 
altitude and speed were correlated (Figure 
1 ). 

More details of the selection, training, and 
evaluation process are given in Ali et al. 
(2003). 



Figure 1: Varaitlon of Altitude & Speed 


Results and Discussion 

A three-way , split plot ANOVA was 
conducted with one within-subjects factor, 
evaluation flight #1 vs. evaluation flight #2, 
and two between-subjects factors, training 
time and feedback condition. There was no 
significant difference in performance scores 
between the two evaluation flights, F(l, 1 5) 
< 1, and interactions between evaluation 
trials and training time, feedback condition, 
and evaluation time by feedback condition 
interaction were all not significant (all F 
values < 1). For the between-subjects 
factors there was no significant interaction 
between training time and feedback 
condition, F (1 , 15) = 2.99. However, there 
was a significant difference in performance 
scores between feedback and No-PFFB 
groups, F (1 , 1 5) = 6.4 1 8, p < .05. 
Specifically, participants in the PFFB groups 
performed better than those in the No-PFFB 
groups in the evaluation flights. In addition, 
there was a significant difference in 
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performance scores for training time, i.e., 
participants in the RTT/RTT groups 
performed better than those in the ARTT 
groups on evaluation trials, F(1 ,1 5) = 7.009, 
p < .05. 

The various training interventions were 
observed to influence performance across 
training flights and the evaluation flights 
(Figure 2). The participants who 
experienced RTT without feedback may be 
viewed as a control group and it may be 
seen that their performance improved with 
practice in the absence of extrinsic feedback; 
however, this improvement did not appear 
substantially until the eighth training 
session. In general, participants in the 
feedback groups improved performance 
early in training, and performed better than 
their counterparts across training sessions. 
However, as can be observed from Figure 2, 
the error in the evaluation flights for both 
the RTT groups was higher than the errors 
during training. This is attributed to the 
increased complexity of the task. The group 


which received ARTT without feedback 
( 1 .5/2.0 No-PFFB) did not exhibit 
improvement with time, and worsened in 
performance after switch over from 1 .5 to 
2.0 ARTT. This perhaps was due to the 
increased difficulty of the task due to the 
switchover to a higher value of ARTT 
conditions. 

It was also observed that the 1 .5/2.0 No- 
PFFB group performed the worst in the 
evaluation flights (Figure 2), exhibiting a 
poorer transfer of training from training 
conditions to evaluation conditions. Perhaps 
responses acquired during the ARTT carried 
over into the evaluation flights in RT. 

EXPERIMENT 2 

In the previous study participants who 
experienced ARTT values of 1.5 followed 
by 2.0 were compared to those who 
experienced real-time training. The purpose 
of this second study was to expand the 
previous study by comparing the previous 



Figure 2 : Average Errors During Training & Evaluation Flights 
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four groups with two additional groups of 
participants who received ARTT at a value 
of 2.0 followed by 1 .5. One group received 
feedback and the other did not. 

Participants, Experimental Design and 

Procedure 

Two groups of five novice pilots in each 
participated, and their data was combined 
with the previous four groups, making a 
total of 30 participants. Thus, with the 
addition of the two new groups, the design 
was a 3 X 2 X 2 mixed factorial design, with 
the ARTT condition ( 1 .5/2.0, 2.0/1 .5, 

1 .0/1 .0) and feedback (PFFB/No-PFFB) as 
the between subjects factors and evaluation 
trial as the within subjects factor. The 
procedure was exactly the same as in the 
previous study. 

Results and Discussion 

Participants in the 2.0/1. 5 ARTT sequence 
exhibited some interesting characteristics. 
The No-PFFB 2.0/1. 5 ARTT group 
demonstrated a continuous improvement in 
performance over the training flights even 
after the switch from 2.0 to 1.5 ARTT. 
However, in comparison to the switch over 
error during training, there was a relatively 
larger error exhibited during the evaluation 
flights which occurred in RT. This again 
suggests that performance in the evaluation 
flights was being affected by both transfer of 
training effects as well as a change in task 
complexity. It is however noted that this 
impact is lesser as compared to the larger 
change of slowing down from 2.0 ART to 
RT as was in the case of the 1 .5/2.0 group. 
The performance by the 2. 0/1. 5 ARTT with 
PFFB group showed the impact of PFFB 
through continued improvement as training 
progressed which is consistent with the other 
PFFB groups. However, its performance 
during the evaluation flights was observed to 
be not as good as the 2.0/1 .5 No-PFFB 
group. This is attributed to the fact that the 
PFFB group was not as strong (the PFFB 
group had an initial average error of 0.25 as 
compared to 0.14 for the No-PFFB group). 
Thus, performance on the evaluation task 


(which was more challenging) was more 
depressed. 

A split-plot ANOVA was carried out 
with evaluation trial as the within subjects 
factor and ARTT condition and feedback 
condition as the between subjects factors. 
There was no significant difference between 
evaluation trials and the interactions 
between evaluation trials and the other 
factors were not significant. The between 
subjects effects were also not significant. An 
analysis of the means of the evaluation 
trials, however, shows that the 1 .5/2.0 No- 
PFFB group performed considerably worse 
than all other groups. 

SUMMARY AND CONCLUSIONS 

In summary, the analysis suggests that for a 
coordinated level turn: 

1 . 1 .5/2.0 ARTT with No-PFFB is not an 
effective training strategy; 

2. ARTT does not seem to offer any 

advantage over RTT for this maneuver; 

3. ARTT should be used in conjunction 

with appropriate PFFB; 

4. PFFB in RTT is the most efficient 

combination. 

Future work should address a) whether the 
depressed performance of ARTT groups 
continues with further evaluation trials, b) if 
pilots trained with one value of ARTT 
throughout training perform better than 
those with RT, and c) whether ARTT is 
beneficial for other flying maneuvers. 
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ABSTRACT 

With the advancements in 
the computing power of personal 
computers, pc-based flight 
simulators and trainers have 
opened new avenues in the 
training of airplane pilots. It 
may be desirable to have the 
flight simulator make a 
quantitative evaluation of the 
progress of a pilot's training 
thereby reducing the physical 
requirement of the flight 
instructor who must, in turn, 
watch every flight. 

In an experiment, 

University students conducted 
six different flights, each 
consisting of two level turns. 
The flights were three minutes 
in duration. By evaluating 
videotapes, two certified flight 
instructors provided separate 
letter grades for each turn. 
These level turns were also 
evaluated using two other 
computer based grading methods . 
One method determined automated 
grades based on prescribed 
tolerances in bank angle, 
airspeed and altitude. The other 
method used was deviations in 
altitude and bank angle for 
performance index and 
performance grades . 

INTRODUCTION 


With the invention of faster 
personal computers it has become 
possible to use flight 
simulation on cost effective 
computers . Because the 
advantages of training on flight 
simulators includes savings in 
time and money, computer based 
training provides researchers 
with increased opportunity to 
investigate such factors that 
may affect flying and pilot 
training. Investigation of 
scoring practices is needed to 
evaluate different strategies in 
training. Computerized scoring 
is desirable as it could, if 
properly implemented, give 
immediate feedback about the 
pilot's performance. 

Straight and level flight, 
climb, descent and level turn 
are regarded as the four basic 
flight maneuvers in a Federal 
Aviation Administration (1995) 
publication on private pilot 
practical test standards. For 
evaluation of a trainee's 
flight, it prescribes certain 
tolerances in the basic flight 
parameters. Following FAA's 
guidelines, an expert flight 
instructor directly evaluates 
the performance of a trainee's 
flight . 

For modern training 
simulators, Vreuls and Obermayer 
(1985) emphasize the need of 
automated performance measures 



as a substitute for the 
evaluation of performance by 
direct observations. They 
present several benefits of the 
automated performance measures. 
The authors have suggested 
several methods for validating 
automated measures . One method 
includes the need of experts to 
judge performance quality, and 
then determine which measures 
correlate with the experts' 
judgments . 

Williams (2000) used grade 
points for flying performances 
on the simulator. To validate 
the automated scoring, several 
flights flown on a simulator 
were simultaneously evaluated by 
the computer and by a certified 
flight instructor. He found 
acceptable correlation for 
straight and level flights but 
low correlation for climb, 
descent and turning maneuvers. 
Ali, Khan, Rossi, Crane, 
Guckenburger and Bageon (2001) 
proposed that a performance 
measure which represents an 
increase or decrease of 
performance at different stages 
of training is a valid measure 
to assess progress in training 
even if the measure is not 
adopted for certifying a trained 
pilot . 

We used an RMS value of 
deviation in bank angle and 
altitude as a performance index. 
We compared the automated 
grades, and performance index 
with the instructors' letter 
grades. Following Williams 
(2000) method, we used grade 
points for our automated grade. 
Furthermore, two instructors 
independently evaluated the 
flights and gave letter grades. 

In an attempt to create a 
numerical grade using the RMS 
value, performance index and 
automated grades were graphed 
and a regression line was drawn. 


University students performed 
six different flights. Each 
flight consisted of two level 
turns. For a level turn task, 
two kinds of performance 
measures were devised, automated 
grades and performance index. 

In the automated grades, A, B, C 
and D grades were based on 
prescribed tolerances in bank 
angle, and altitude. A 
performance index was based on 
deviations in bank angle and 
altitude. Video recordings were 
made of the HUD and relevant 
instrument panel gauges . Two 
certified flight instructors 
independently evaluated the 
videos of the flights. This 
study attempts to correlate the 
performance index, the automated 
grades and instructors' 
evaluations . 

EQUIPMENT 

The experiment was 
performed in the Flight Vehicle 
Design Lab at Tuskegee 
University. Flights were 
conducted on a training 
simulator that had LiteFlite 
version 3.3 installed. The PCs 
for the simulator were two Heavy 
Metal computers made by Quantum 
3D. Each computer has 2 
processors running at 400Mhz, 
400MB RAM, and has three extra 
video cards for the Out of the 
Window (OTW) view. Three 
display monitors showed the OTW 
view. The center monitor 
displayed not only a forward 
view, but also a heads up 
display (HUD) . Figure 1 shows 
the entire setup including the 
four monitors and a moving map 
display monitor on the right of 
the picture . The moving map 
display was not used in this 
experiment . Figure 2 shows a 
close up of the inside the 
cockpit view screen. The 
joystick was a Saitek X36F and 
throttle was Saitek X35T 
controller. Rudder pedals used 
were from CH products. Williams 



(2000) gave more details on the 
fidelity of the equipment used. 




Figure 1 : Simulator setup 


Figure 2 . Instrument Panel 
(Inside the cockpit view) 


METHOD 

Level turn task 

This task was to be a level 
turn from 0 degrees to 180 
degrees heading. The pilot, in 
this scenario, was to bank the 
airplane 30 degrees to the right 


or left while flying at a speed 
of 90 knots. After the plane 
reached 180 degrees, the pilot 
was to bank in the same or 
opposite direction at a new bank 
angle of 45 degrees. The pilot 
was to hold the bank angle and 
speed until the airplane reached 
360 degrees. After that time 
the pilot was to fly straight 
and level until the simulator 
stopped. 

Each turn task was recorded 
by videotaping the HUD on the 
center monitor and the turn and 
bank instruments on the inside 
the cockpit monitor. 


Instructors' grades 

Two Certified Flight 
Instructors were then asked to 
view the videotaped sessions and 
write down a score (A,B,C,D,E or 
F) for, in turn, altitude, bank 
angle and airspeed over the time 
slots: 21-72 seconds, and 
approximately 120 - 171 seconds. 
These time intervals represented 
each of the two turns in a 
flight. An average of the 
three -parameter grades was 
calculated to represent the 
instructors' grade for a single 
turn. This was used as a data 
point. For the six flights 
there were 12 data points 
representing all turns. 

Automated grades 

The automated grade 
criteria were based on the 
requirements of 90 knots +3knots 
in airspeed, 30 or 45 degrees ± 
3degrees in bank angle and 10000 
ft ± 50 feet in altitude. Flight 
parameters within these limits 
were graded as 'A' . Deviations 
of + 6 knots in airspeed, + 6 
degrees in bank angle and + 100 
feet in altitude were graded as 
'B' . Deviations of ±9 knots in 
airspeed, +9 knots in bank angle 
and +150 knots in altitude were 
graded as 'C' . Deviations of +12 
knots in airspeed, ±12 degrees 


5 




in bank angle and +200 feet in 
altitude were graded as 'D' . 

And, deviations of < - 12 knots 
and > +12knots in airspeed, -12 
degrees and >+12 degrees in bank 
angle and < -200 feet and > +200 
feet in altitude were graded as 
'F' . The flight parameters were 
grading every three seconds . An 
average grade was then 
calculated for each of the 
turns . 

Performance Index and 
Performance Grades 

The performance index of the 
level turn flights were obtained 
by taking the root of the 
squared sum of the average 
deviations of the heading, 
altitude, and airspeed. 

It should be noted that as a 
performance index (P.I.) 
increases, the performance of a 
pilot actually decreases. For 
that reason, in order to obtain 
some reasonable 4 point scale 
score we used an equation of the 
form A/ (B+P . I . ) and the values 
tried for A and B were such that 
B= 0 . 25*A. 

RESULTS AND DISCUSSION 

Our set of evaluation 
consisted of 12 data points 
representing 6 flights with 2 
turns in each flight. Five such 
sets of evaluation were: 

Automated grades, instructor 1 
and instructor 2 grades, 
performance index and 
performance grade. A comparison 
of the different evaluations 
that were made is as follows. 


expected, as the subjects were 
novice pilots. However, the two 
instructors' scores were 
somewhat inconsistent with one 
another. In general, 
instructor#l graded higher in 
comparison with the automated 
grade, while the grading of 
instructor#2 was lower than the 
automated grade . 



Figure 3 . Scatter plot of 
Instructor One Grade vs. 
Automated grade (Turn Task) 



Automated grade 


Automated Grade versus 
Instructors' grades 

The comparison of the overall 
automated grade vs . the 
instructors' grades for each of 
the turns is shown as scatter 
charts in Figs. 3 and 4. As can 
be seen from the scatter plots, 
most of the scores are in the 
2.0 or below range. This was 


Figure 4 . Scatter plot of 
Instructor Two Grade vs. 
Computer Automated Grade (Turn 
Tasks) 
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Figure 5. Scatter plot of 
Automated Grade (AG) vs . 
Performance Index (PI) (Turn 
Task) 


Figure 7 . Scatter plot of 
Instructor #2 Grade vs. 
Performance Grade (Turn Task) 


There exists a negative 
correlation of the automated 
score with the performance 
index. This should be expected 
as the performance index has an 
inverse relationship with both 
the instructor grades and the 
automated score. However, the 
best-fit regression more 
resembles a curve of the form: 

AG = 0.5265/ > / A (-1.0198) (Figure 
5) . 



Figure 6. Scatter plot of 
Instructor #1 Grade vs. 
Performance Grade (Turn Task) 



Automated Grade 


Figure 8. Scatter plot of 
Performance Grade vs . Automated 
Grade (Turn Task) 

In order to obtain some 
reasonable 4 point scale score 
we used an equation of the form 
A/(B+P.I.) and the values tried 
for A and B were such that B= 

0 ,25*A. 

The formula: 1.0/ (0.25 + P.I.) 
seems to be better suited for 
the purpose of determining 
performance grade from 
performance index. This formula 
was used after several attempts 
with other similar formulae. 

Most of the P.I. values seemed 
to match with 0.25. 


The performance grades appear to 
correlate well with the 
automated grades and the 
automated grades correlate well 
with the averaged instructor 
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grades. The instructor #l's 
grades correlate better with the 
performance grade than does 
instructor #2 's grades. In 
general instructor #2's grades 
don' t seem to correlate well 
with any of our computer grading 
measures . 

This analysis suggests 
that letter-grades may not be 
appropriate for tracking 
training progress as for novices 
the improvements are 
incremental . Thus they may 
exhibit improvements in 
controlling individual flight 
parameters but still the overall 
'letter' grade may not show an 
improvement in overall skill . 
Although the correlations are 
moderate to high between 
instructors, the instructors 
both seem to have a systematic 
difference between their score 
and the automated grade . 

Further, the differences in 
correlations on different 
parameters between instructors 
and the computer grade suggests 
that perhaps, the instructors 
are making evaluations based on 
different factors from each 
other. Thus the factors 
influencing the assessments of 
the instructors need to be 
understood. Future studies could 
systematically address these 
issues . 

A failed attempt was made for a 
power regression between the two 
instructors grades and the 
performance index. 

An increase in performance 
grades for a level turn 
corresponds with an increase in 
performance for a novice pilot. 
Therefore, they can be 
considered for further study for 
the evaluation of progress in 
training of level turns in 
simulators . 


CONCLUSION 

Four kinds of grading have 
been obtained and compared for 
evaluating level turn flights 
performed by novice pilots in a 
simulator. They are: 
instructor's grades, automated 
grades, performance index and 
performance grades. 

Instructor's grades are the 
grades provided by a certified 
flight instructor. Automated 
grades are based on prescribed 
tolerances in bank angle, 
altitude and airspeed. The 
performance index (P.I.) is 
created using the square root of 
the sum of the squares of 
dimensionless deviations in 
altitude and bank angle. 
Performance grade, which is 
defined as 1/(0. 25 + P.I.), 
provides a reasonable 4 -point 
grading scale. The comparison 
of performance grades with the 
other three performance 
measures, leads to the 
suggestion that they deserve 
further study for the evaluation 
of progress in training of level 
turns in simulators. 
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